Skip to content

Cross compiling using github actions

Jump to bottom
David O'Rourke edited this page Mar 19, 2020 · 1 revision

Cross compiling a vcvrack plugin using github actions

abstract

Previously I had the luxury of being able to do non-work stuff on my work PC. This meant that I could compile my plugins for Microsoft Windows at work, while compiling them for linux at home. That's not been an option for a while, as my only development environment is Ubuntu 16.04 at home.

Getting a plugin into the vcv library means that it will be automatically built and zipped for all three of the supported platforms, but it usually takes a few days for the changes to be reviewed and built, sometimes users are asking for platform specific builds sooner. Or you might want to put a beta out there for people to look at, without committing that to the library. There are usually people on the forums who are willing to compile a plugin for you, but now that GitHub provides GitHub actions for CI/CD; I thought I would try to use their virtual machines to get platform specific builds done on my own terms.

In this article I discuss what I have working in my projects, how they do their job, and what options you have if you want to do something similar, but perhaps not identical.

Not all of this work is my own. Obviously I've relied on other people's examples to get to grips with the GitHub actions syntax, and the GitHub api. Most importantly, the Docker images that I use came from dewb following a suggestion that he made on the VCVRack community forums. I wouldn't have got anywhere without those. Thank you.

background

GitHub actions are CI/CD tools. They provide virtual machines and script runners that can perform build, test and other operations on the contents of your repositories. Typically they start automatically in response to certain triggers, and they run non-interactively until they either complete successfully, or get stopped by something like a compilation error.

GitHub Actions Documentation

getting started

Typically GitHub workflows live in your own repository in the /.github/workflows directory, and they make use of actions which might be in your repository under /.github/actions they might be published by third parties and be available in the GitHub marketplace.

I built a workflow that is triggered when I publish a release. It builds a distribution for each of the three supported platforms, and adds each distribution zip as a binary file to the newly published release. It then takes those three zip files and repackages them as a single multi-platform zip file.

There are lots of different triggers available, workflows can be run after every check-in, or after completion of pull requests. But they can also be run when people create, update, or close issues; or all kinds of other events.

You might prefer to create a release and build automatically after every check-in, or merge, or PR. This article doesn't cover those specifics, but it might give you an idea of where to start.

workflow

The workflow is written in YAML and lives in a file in the /.github/workflows directory of your repository. You can give it any sensible name, what it does and when it does it, is defined within the workflow file itself; when an event occurs in a repository, GitHub will check the workflows and see if they should be run.

Note that because workflows are in a repository, they are versioned.

Here is my buildRelease.yml file. A workflow which does all the work to build the distributions and attach them to the release:

on:
  release:
    types: [published]
name: Release
env:
  RACK_SDK_VERSION: 1.1.6
jobs:
  buildLinux:
    name: Build Linux
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@master
    - name: Build Linux
      uses: ./.github/actions/build_linux
    - name: upload zip
      run: sh ./.github/actions/upload_zip/script.sh ${{ secrets.GITHUB_TOKEN }}
  buildWindows:
    name: Build Windows
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@master
    - name: Build Windows
      uses: ./.github/actions/build_win
    - name: upload zip
      run: sh ./.github/actions/upload_zip/script.sh ${{ secrets.GITHUB_TOKEN }}
  buildOsx:
    name: Build OSX
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@master
    - name: Build OSX
      uses: ./.github/actions/build_osx
    - name: upload zip
      run: sh ./.github/actions/upload_zip/script.sh ${{ secrets.GITHUB_TOKEN }}
  combineDist:
    name: Combine Distributions
    runs-on: ubuntu-latest
    needs: [buildLinux, buildWindows, buildOsx]
    steps:
    - uses: actions/checkout@master
    - name: combine zip
      run: sh ./.github/actions/combine_zip/script.sh ${{ secrets.GITHUB_TOKEN }}

Let's break it down:

head

on:
  release:
    types: [published]
name: Release
env:
  RACK_SDK_VERSION: 1.1.6

At the top we have the triggers for this workflow. The workflow will run when a release event occurs, and we have specifically filtered that to a subset of just published. So this will run when I publish a release; either by changing the state of a release to publish, or I create it in the published state.

We give the workflow a name Release which is the internal name that the GitHub actions system will know it by. The actual file name of the yaml file is irrelevant to the workflow engine, this internal name is important.

Finally in this section I have defined an environment variable which I will use in several steps. Environment variables can be also be defined per job or per step if they have a more limited scope.

jobs

jobs:
  buildLinux:
    name: Build Linux
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@master
    - name: Build Linux
      uses: ./.github/actions/build_linux
    - name: upload zip
      run: sh ./.github/actions/upload_zip/script.sh ${{ secrets.GITHUB_TOKEN }}

The jobs section of the file lists the jobs that we want to run. Each job runs independently in it's own virtual machine. By default they can run in parallel. There are 4 jobs in this workflow, called buildLinux, buildWindows, buildOsx and combineDist. I let the first 3 run in parallel, and then we'll see later that combineDist waits for the first 3 to complete before starting.

Each job has a job_id in the yaml file e.g. buildLinux and it also has a display name given by the name property; e.g. Build Linux

The runs-on property tells GitHub which virtual environment to run it on. GitHub provides a handful of different operating system environments including versions of Ubuntu, Microsoft Windows and macOS

The job is then broken down into steps, each of which is run sequentially. By default if any step fails, the workflow job will fail.

Each step can have a display name. In this example I've not given a name to the first step. That first step is an action in a public repository. The uses property specifies the action as actions/checkout and also that we should use the version of actions/checkout from the head of master branch. An action specification will usually include some sort of version specifier, which could be a branch, tag or commit id. For a publicly available action like this, the author will usually recommend which version to use. The actions/checkout@master action downloads a repository into a working directory in the virtual machine; by default it will download the version of the repository associated with the event that triggered the workflow. So in this case where I am triggering the workflow on the publication of a release, it will be the version of the repository whose tag is associated with that release.

There are two further steps in this job and these are both non-public actions local to this repository. I'll tackle them one at a time.

'Build Linux' action

The Build Linux action, found in the directory /.github/actions/build_linux is a Docker action and I am using it with the uses property as I did for the checkout action. I have not specified a version for it, because I'm not asking GitHub to retrieve it from a public repository. Instead I'm specifying the path to it. The checkout action has already pulled it down into the working directory structure.

Looking a the /.github/actions/build_linux directory we can see two files, a Dockerfile and a shell script. The presence of the Dockerfile is what informs GitHub as to what should be done with this action. I've simply specified the directory in the uses property, and GitHub is having to figure out what is necessary.

FROM ubuntu:16.04

LABEL "com.github.actions.name"="VCVRackPluginBuilder-Linux"
LABEL "com.github.actions.description"="Builds a VCV Rack plugin for Linux"
LABEL "com.github.actions.icon"="headphones"
LABEL "com.github.actions.color"="purple"

LABEL "repository"="TBD"
LABEL "homepage"="TBD"
LABEL "maintainer"="dewb"

RUN apt-get update
RUN apt-get install -y build-essential cmake curl gcc g++ git make tar unzip zip libgl1-mesa-dev libglu1-mesa-dev jq

ADD entrypoint.sh /entrypoint.sh
RUN chmod a+x /entrypoint.sh

ENTRYPOINT ["/entrypoint.sh"]

Using Docker for this is not strictly necessary, but it provides a certain level of isolation. If github decide to change the version of compilers or tools installed on the virtual machine, that might break my build. By using Docker, I'm specifying that my build should take place in a container in that virtual machine, and I've specified that the tools I want to use are in the ubuntu:16.04 Docker image.

Docker will create a container using the ubuntu:16.04 image, then install some extra tools using apt-get It will map the entrypoint.sh file from my action directory to an entrypoint.sh file in the root of the container, make it executable using chmod and then run that entrypoint shell script.

#!/bin/sh

set -eu

export RACK_DIR=${GITHUB_WORKSPACE}/Rack-SDK
export RACK_USER_DIR=${GITHUB_WORKSPACE}

git submodule update --init --recursive

curl -L https://vcvrack.com/downloads/Rack-SDK-${RACK_SDK_VERSION}.zip -o rack-sdk.zip
unzip -o rack-sdk.zip
rm rack-sdk.zip

make clean
make dist

The shell script will set up a couple of local environment variables, using the GITHUB_WORKSPACE environment variable provided by the workflow runner. It will update any necessary submodules in the checked out repository. Then it will download and unzip the Rack-SDK. Note that it uses the global environment variable RACK_SDK_VERSION that we defined at the top of the workflow yaml. Finally it will make a distribution of the plugin using make.

If I actually had any submodules in my project, then I should probably do a make dep as part of that script.

'Upload Zip' action

The Upload Zip action found in the /.github/actions/upload_zip directory is not called from the yaml file as an action, because it contains purely a shell script, I have just explicitly run that script using the line in the yaml

run: sh ./.github/actions/upload_zip/script.sh ${{ secrets.GITHUB_TOKEN }}

Note that a secrets.GITHUB_TOKEN is passed in. This is a token for the GitHub api. It is automatically generated by the workflow runner, and by using it as an authentication token when the shell script POSTS the built zipfile to the release, that it has permissions to do so.

The shell script is below:

#!/bin/sh

set -eu

GITHUB_API_URL=https://api.github.com

GITHUB_TOKEN=$1

# Get release url
curl -o release.json \
    --header "Authorization: token ${GITHUB_TOKEN}" \
    --request GET \
    ${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/releases/${GITHUB_REF#"refs/"}

UPLOAD_URL=$(jq -r .upload_url release.json)

ASSET_PATH=$(ls dist/*.zip)

curl -i \
    --header "Authorization: token ${GITHUB_TOKEN}" \
    --header "Content-Type: application/zip" \
    --request POST \
    --data-binary @"${ASSET_PATH}" \
    ${UPLOAD_URL%"{?name,label\}"}?name=${ASSET_PATH#"dist/"}

The script uses curl to call the GitHub api to get details of the release that I just published. The GITHUB_REF environment variable contains the ref relevant to the event that triggered the workflow.

I use jq to extract the upload url from the api reponse.

Then I can make a second call to the api to post the zip file onto the release.

building the other platforms

The second and third jobs work in an almost identical fashion to build the distribution for Microsoft Windows and for macOS. In both cases I have used a linux virtual machine, a linux Docker image and cross compilation tools to do the building. It would be entirely possible to do native compilation using the Microsoft Windows and macOS virtual machines available to GitHub actions, but if you are using a professional GitHub subscription, you need to pay for the use of the virtual machines, and the linux machines are cheaper.

combining the distributions

The final job in the yaml file combines the three distribution zipfiles into a single multi-platform distribution.

 combineDist:
    name: Combine Distributions
    runs-on: ubuntu-latest
    needs: [buildLinux, buildWindows, buildOsx]
    steps:
    - uses: actions/checkout@master
    - name: combine zip
      run: sh ./.github/actions/combine_zip/script.sh ${{ secrets.GITHUB_TOKEN }}

The needs property indicates that this job is dependent on the three earlier build jobs. Although the three build jobs will run in parallel, this combine job will not start until all the builds have completed. Again we checkout the repository using the actions/checkout@master action. Then we run a shell script to do the work.

#!/bin/sh

set -eu

GITHUB_API_URL=https://api.github.com

GITHUB_TOKEN=$1

curl -o release.json \
    --header "Authorization: token ${GITHUB_TOKEN}" \
    --request GET \
    ${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/releases/${GITHUB_REF#"refs/"}

UPLOAD_URL=$(jq -r .upload_url release.json)

DOWNLOAD_LIST=$(jq -r ".assets | .[] | .url" release.json)

rm release.json

mkdir dist

echo "$DOWNLOAD_LIST" | while IFS= read -r line 
do 
	curl -L -o $(basename $line).zip --header "Authorization: token ${GITHUB_TOKEN}" --header "Accept: application/octet-stream" --request GET $line
	unzip -o $(basename $line).zip -d dist
	rm $(basename $line).zip
done

PLUGIN_BUILD_SLUG=$(jq -r .slug plugin.json)
PLUGIN_BUILD_NAME=${PLUGIN_BUILD_SLUG}-$(jq -r .version plugin.json)
cd dist 
zip -q -9 -r ${PLUGIN_BUILD_NAME}.zip ./${PLUGIN_BUILD_SLUG}
cd ..

ASSET_PATH=$(ls dist/*.zip)

curl -i \
    --header "Authorization: token ${GITHUB_TOKEN}" \
    --header "Content-Type: application/zip" \
    --request POST \
    --data-binary @"${ASSET_PATH}" \
    ${UPLOAD_URL%"{?name,label\}"}?name=${ASSET_PATH#"dist/"}
  • Once again we use curl to retrieve the details of the release from the GitHub api.
  • extract the upload url, and a list of existing assets (the three previously uploaded zipfiles)
  • create a dist directory
  • download each of the existing assets and unzip them into the dist directory
  • extract information from the plugin.json file in our plugin repository
  • make a new zipfile containing the assets and all three executable files from the dist directory
  • finally we upload that new zipfile also onto the release.

conclusion

This is just one example of a workflow, and is very specific to my desire to build all the platform specific builds of my plugin. All of the necessary files can be found in my repository.

Thanks once again to dewb for the head-start

Quick Links

Torpedo Modem Protocol

Technical Articles
Cross-compiling using actions
Examining the sine approximation
Refactoring the approximation
SSE random number generation
Smooth Noise SN-101

Manual

Releases

Clone this wiki locally