Packaging xLights for linux as AppImage

Now that I have xLights being built fine within docker (see Automatic testing of xLights builds via Travis-CI/Docker/Github) the next step is to package up the application as an appimage binary so that it can easily be run on varied linux systems.

It is being built based on Ubuntu Trusty and has been tested to work on Ubuntu 16.04, 16.10, Fedora 25 and OpenSuse 42.2 (and should hopefully work pretty much anywhere else as well)

The creation of the AppImage is done by the Recipe.appimage file that I included in the docker image.  This means that building of the appimage is simply:

$ docker pull debenham/xlights
$ docker run --name buildvm debenham/xlights /bin/bash Recipe.appimage

I’ll break down the Recipe now to explain how it works:

First up we run the Recipe script to build xLights (as shown in previous post).

#!/bin/bash
./Recipe

Next up we setup the environment ready for later on.  The $VERSION number is generated using the version number in the xLights_4_64bit.iss (which is where the version number is explicitly set for the windows install and so is a good/safe place to grab from).   If I am generating a release version (by passing ‘release’ as the command-line option) it will leave it at that.  If this is not a release version (such as for testing/development) then it will also add the short hash of the last commit.  This is handy so I can easily tell which commit the package was built from.

export BASEDIR=/xLights
cd ${BASEDIR}/xlights-git
APP=xLights
COMMIT=`git log --pretty=format:'%h' -n 1`
VERS=`grep AppVersion xLights_4_64bit.iss |sed -e 's/^.*=//g'`
if [ "$1" = "release" ]
then VERSION=${VERS}
else VERSION=${VERS}-${COMMIT}
fi

Now we install xLights into the target AppDir directory

rm -rf ${BASEDIR}/$APP/
 mkdir ${BASEDIR}/$APP/
 make install DESTDIR=${BASEDIR}/$APP/$APP.AppDir PREFIX=/usr

Okay, we have xLights installed in ${BASEDIR}/$APP/$APP.AppDir so the next step is to setup this directory ready to be packaged up.  To do this we import the functions.sh script which contains a bunch of handy functions needed for building AppImages

cd ${BASEDIR}/$APP/

wget -q https://raw.githubusercontent.com/AppImage/AppImages/master/functions.sh -O ./functions.sh
 . ./functions.sh

Lets put the appicons/desktop launcher in place ready to go

cd ${BASEDIR}/$APP/$APP.AppDir

cp usr/share/icons/hicolor/256x256/apps/xlights.png .
cp usr/share/icons/hicolor/256x256/apps/xschedule.png .
cp ./usr/share/applications/xlights.desktop .

Next we need to grab the AppRun binary (which is basically a wrapper which takes care of setting up all the environment so the binary inside the AppImage uses the right libraries/paths etc).   Since xLights needs a newer libstdc++.so.6 than is included in Ubuntu Trusty we needed a specially patched AppRun which allows for the bundling of the newer library without breaking systems which already have a newer library.  See https://github.com/darealshinji/AppImageKit-checkrt/ for details of why this is needed.

wget -O AppRun https://github.com/darealshinji/AppImageKit-checkrt/releases/download/continuous/AppRun-patched-x86_64
 chmod +x AppRun

Now we need to find all the libraries needed by xLights and put them in place.  This is handled automatically by the copy_deps function we imported previously from functions.sh.  I have to manually move the pulseaudio libraries to the common directory (so AppRun doesn’t need to be modified further) and also remove libharfbuzz.* as it causes xLights to be unable to run if it is included.  I also manually copy libstdc++.so.6 to the special path as needed by the modified AppRun.  We finish up by stripping the libraries to save space.

export LD_LIBRARY_PATH=./usr/lib/:$LD_LIBRARY_PATH

copy_deps ; copy_deps ; copy_deps # Three runs to ensure we catch indirect ones
 move_lib
 mv usr/lib/x86_64-linux-gnu/pulseaudio/* usr/lib/x86_64-linux-gnu
 rm usr/lib/x86_64-linux-gnu/libharfbuzz.*
 mkdir -p usr/optional/libstdc++
 cp /usr/lib/x86_64-linux-gnu/libstdc++.so.6 usr/optional/libstdc++/
 delete_blacklisted

strip usr/lib/* usr/lib/*/* || true

 

Almost done now. Just tell AppImageKit what the target launcher will be

get_desktopintegration xlights

Final steps now.  First we check if we are running within Docker. This is needed so that the generate function knows to not use FUSE to mount the image (since FUSE doesn’t work properly within a standard docker container)

cd ..

########################################################################
 # AppDir complete
 # Now packaging it as an AppImage
 ########################################################################

if [[ ! $(cat /proc/1/sched | head -n 1 | grep init) ]]; then {
 echo in docker
 DOCKER_BUILD="yes"
 } fi

And the very last step is to call generate_type2_appimage to actually create the appimage file itself

generate_type2_appimage

After all this is done we are left with a xLights.xxxxxx.AppImage file sitting in ${BASEDIR}/out – ready for uploading to the web and people to use!

For me I then scp them to my web server and wordpress will show the new file automatically at https://www.adebenham.com/xlights-linux/

Automatic testing of xLights builds via Travis-CI/Docker/Github

xLights is a free and open source program that enables you to design, create and play amazing lighting displays through the use of DMX controllers, E1.31 Ethernet controllers and more.

With it you can layout your display visually then assign effects to the various items throughout your sequence. This can be in time to music (with beat-tracking built into xLights) or just however you like.
Currently xLights runs on Windows, OSX and Linux

I look after the Linux build and since most of the other core developers are running Windows or OSX build/compile issues occasionally come up.

To help know when/how such breakage occurs I have setup automatic builds via travis-ci which are triggered after every commit.  This way when a commit breaks the linux build I get an email so I can quickly go and check/fix as needed.  Thanks to the integration with GitHub

Setting this up was a bit more complicated than just turning on builds in travis-ci.

By default the build environment on travis-ci is Ubuntu trusty (which is the Ubuntu I want to use as it is the oldest LTS still supported and so useful for AppImage builds) – but sadly that doesn’t have new enough libraries (in particular the ffmpeg libraries) and also the build toolchain is too old (need C++14 support which is only in the newer gcc).  Adding these dynamically every build was not practical.  As such I had to build a custom docker image which was pre-configured with the necessary libraries/tools etc.  Doing it this way also meant that I could provide a pre-compiled wxwidgets as part of the image rather than having to rebuild it every time (this saves substantial time).

To build the Docker image I had to create a suitable ‘Dockerfile‘ which describes the build of the image.  This dockerfile was as follows:

FROM ubuntu:14.04

ADD cbp2make /usr/bin/cbp2make
ADD Recipe.deps /Recipe.deps
ADD Recipe /Recipe
ADD Recipe.appimage /Recipe.appimage

RUN bash -ex Recipe.dep

This dockerfile is different to most others I have seen.

It starts by saying the base image (Ubuntu 14.04) and then adds a couple of files.  Rather than having the dockerfile cause the complete build of xLights it just sets up for future builds.

It adds four files, the first being a quick pre-built cbp2make (since that is not available in ubuntu 14.04 – this was the simplest way to add)

The next three are Recipe files.

  • Recipe.deps just sets up the image by installing dependencies and pre-building wxWidgets.
  • Recipe clones the source of xLights from GitHub and then builds it.
  • Recipe.appimage takes the built xLights and packages it up as an AppImage binary.

Then it only runs the Recipe.deps – this way the docker image is ready to build and we don’t waste time rebuilding things that don’t change.

Once this was setup with all those files in a single directory the next step was to build the docker image and upload to docker hub.

$ ls
cbp2make Dockerfile out Recipe Recipe.appimage Recipe.deps
$ docker build -t xlights_image .
$ docker create --name xlights_container xlights_image
$ docker tag xlights_image debenham/xlights:latest
$ docker push debenham/xlights:latest

This simply build the docker image and called it ‘xlights_image’.  This runs the Dockerfile script which adds the required files and then runs Recipe.deps to preconfigure the image.

It then created a container from this image, tagged it with my dockerhub username and finally uploaded to dockerhub so that Travis-CI can use it.

At this point I could test the build (from almost anywhere) by simply pulling the image and running the ‘Recipe’ script.

$ docker pull debenham/xlights
$ docker run --name buildvm debenham/xlights /bin/bash Recipe

If I want to build the AppImage binary I do the same thing – but run the Recipe.appimage script instead.

$ docker pull debenham/xlights
$ docker run --name buildvm debenham/xlights /bin/bash Recipe.appimage

Now all that was left was to integrate this with GitHub/Travis-CI.  This is done by creating a ‘.travis.yml‘ file in the root of the git repository.

Thanks to the docker image this file is simple

dist: trusty
sudo: required
services:
    - docker
git:
    depth: 3
script:
    - docker pull debenham/xlights
    - docker run --name buildvm debenham/xlights /bin/bash Recipe
notifications:
    email:
      recipients:
        - chris@adebenham.com
        - keithsw1111@hotmail.com
     on_success: never
     on_failure: always

This travis file starts by saying what the build needs (ubuntu trusty, docker, sudo etc).  Then describes how to build/test xLights (by pulling the image and then running the Recipe).  Once this is done it says what sort of notifications are needed.  In this case it will notify via email to myself and one other developer.  It also says that want to be notified when the build fails (rather than on every build)

All that remained was going to Travis-CI and turning on the automatic build.

In a coming post I will describe how I build both AppImage binaries and Ubuntu packages for each release.