Build your own slimmer liquidsoap from opam

[vitoyucepi]

Disclaimer

This guide covers building a minimal Liquidsoap installation from source. It involves advanced configuration of OPAM, Dune and Liquidsoap's build system. Significant experience with Linux build tools is assumed.

The user has full responsibility for verifying and testing the resulting Liquidsoap build. A thorough validation in a non-production environment is highly recommended before deployment.

While this guide is intended to provide a working minimum configuration, the inherent complexity means that there is a significant risk of errors preventing proper functioning. Proceed at your own risk - there is no warranty of fitness for a particular purpose.

Why?

The official liquidsoap release artifacts are great and versatile. However, sometimes you may want to build a minimal version with only the essential features to reduce dependencies and size. In this post, I'll show you how to build a minimal Liquidsoap using the OCaml package manager OPAM.

Prerequisites

For this build, I will be using an Alpine Linux container image. Using Alpine allows me to minimize the overall size of the container image. This is beneficial for my goal of building a minimal liquidsoap installation.

Note

The guide will work more or less the same in other distros, but you'll have to tweak the package manager commands.

Preparing the Environment

Inside the Alpine container, I first need to install some binary dependencies:

apk add \
  alpine-sdk \
  opam \
  bash \
  rsync;

Next, I need to create an OPAM switch with the ocaml compiler:

opam init --bare --no-setup --disable-sandboxing;
opam switch create liquidsoap ocaml-base-compiler.4.14.1;

This will set up a minimal environment for building Liquidsoap.

Note

I am using ocaml 4.14.1 because it is the latest ocaml LTS release.

I also need to install dune to configure the build process:

opam install dune;

Now I have the essential dependencies in place to build Liquidsoap.

Getting the Source

Liquidsoap uses OPAM for its build system. You can either install it from opam or build from release artifacts.
In this guide I'll describe the last one, because it can be useful for building custom versions or applying patches.

Note

There are two GitHub repositories with liquidsoap artifacts.

• https://github.com/savonet/liquidsoap
  The main repository provides source code in addition to release artifacts.
• https://github.com/savonet/liquidsoap-release-assets
  The release assets repository provides immutable links to release artifacts.

First, I'll find the release version I want, download it, and check the checksum:

LIQUIDSOAP_VERSION=2.2.3
LIQUIDSOAP_RELEASE="https://github.com/savonet/liquidsoap-release-assets/releases/download/v${LIQUIDSOAP_VERSION}"
LIQUIDSOAP_ARCHIVE="liquidsoap-${LIQUIDSOAP_VERSION}.tar.gz"
LIQUIDSOAP_SHA256SUM='064726d76ef3ad62e622af595d4c5b5603151fdd9f4c0d1f5ec2eab8f4f2f39a'
wget -O liquidsoap.tar.gz "$LIQUIDSOAP_RELEASE/$LIQUIDSOAP_ARCHIVE";
echo "$LIQUIDSOAP_SHA256SUM  liquidsoap.tar.gz" | sha256sum -c;

This ensures that I get the correct source archive from the Liquidsoap project without any corruption.

I can now extract the source into a directory:

mkdir liquidsoap-src;
tar -xzf liquidsoap.tar.gz -C liquidsoap-src --strip-components 1;
cd liquidsoap-src;

Now I have the stable Liquidsoap source code ready to build from.

Installing Build Dependencies

Now that I have extracted the Liquidsoap source code, I need to pin the included packages and install the dependencies from OPAM.

First, I'll pin the packages to the versions included in the source:

opam pin \
  --no-action \
  --yes \
  .;

Next, I can install the OPAM dependencies for these packages:

opam install \
  --deps-only \
  --confirm-level=unsafe-yes \
  liquidsoap-lang \
  liquidsoap-libs \
  liquidsoap-libs-extra \
  liquidsoap-core \
  liquidsoap;

With these dependencies from both APK and OPAM, I now have everything I need to build Liquidsoap.

Configuring the Build

Now that essential dependencies are installed, I will install some non-essential ones that are still useful:

opam install \
  --confirm-level=unsafe-yes \
  ffmpeg \
  yaml;

This adds:

• ffmpeg - For additional media codecs.
• yaml - For yaml file parsing.

These are not required for a minimal build, but provide useful functionality.

Next, load the OPAM environment variables:

eval $(opam env);

This will set the necessary vars like PATH, CAML_LD_LIBRARY_PATH, etc. In addition to setting variables, it allows me to use the dune command that I installed via OPAM in the previous step.

Then export the build target:

export LIQUIDSOAP_BUILD_TARGET=posix;

This sets the build target to posix instead of the default. The default target places libraries and binaries in the OPAM directory structure. Overriding to posix will use system paths like /usr/lib instead.

I will also export a variable to configure the build metadata:

export IS_SNAPSHOT=false;

Setting IS_SNAPSHOT to false will cause the Liquidsoap version to be displayed without any +dev suffix. This results in a clean version output that indicates a release build rather than a development snapshot.

Building

First, compile everything with optimizations enabled:

dune build --release;

This will build all the libraries and binaries in release mode.

Then I can install all the required packages to a desired path:

dune install --relocatable \
  --prefix "/liquidsoap/usr" \
  --docdir="$PWD/.omit" \
  --libdir="$PWD/.omit" \
  liquidsoap-libs \
  liquidsoap-libs-extra \
  liquidsoap-core;

This will:

• Install to /liquidsoap/usr instead of the default OPAM location.
• Omit the docs and lib directories to minimize size.
• Only install the core libraries and binaries needed.

Next, rename the lang directory:

mv /liquidsoap/usr/share/liquidsoap-lang /liquidsoap/usr/share/liquidsoap;

This is required for Liquidsoap to find its standard libraries. Without it, Liquidsoap would fail to start with an error.

Next, copy the Camomile data:

cp -r "$(opam var share)/camomile" /liquidsoap/usr/share/liquidsoap;

This copies the necessary Camomile data for text processing.

This completes the main build process. The compiled Liquidsoap binary and libraries are located in the /liquidsoap directory. To use this minimal Liquidsoap installation, it should be copied to the / filesystem.

Copying to Production

I have built minimal Liquidsoap in the build stage container at /liquidsoap.

To use this in production, I need to copy it to the next stage:

FROM alpine AS build
# Build steps...

FROM alpine AS production
COPY --from=build /liquidsoap /

This will copy the /liquidsoap directory from the build stage into / in the production stage.

This brings over the Liquidsoap artifacts, but does not include any dependencies.
To identify required libraries:

ldd /usr/bin/liquidsoap 2>&1 | grep 'No such file or directory';

This outputs missing dependencies.

I can install them in production:

apk add --no-cache \
  so:libjemalloc.so.2 \
  so:libswresample.so.5 \
  so:libswscale.so.7 \
  so:libavdevice.so.60 \
  so:libavutil.so.58 \
  so:libavformat.so.60 \
  so:libavfilter.so.9 \
  so:libavcodec.so.60 \
  so:libcurl.so.4 \
  so:libpcre.so.1;

By inspecting and installing dependencies manually, the production environment is fully functional.

[smimram]

Thanks a lot! Maybe could this be included somewhere on the official doc?

[toots]

This looks great indeed! We should add it to the doc.

[gAlleb]

Hi @vitoyucepi. I've tried to run your steps but I get

 => CACHED [build 14/20] RUN eval $(opam env);                                                                              0.0s
 => CACHED [build 15/20] RUN export LIQUIDSOAP_BUILD_TARGET=posix;                                                          0.0s
 => CACHED [build 16/20] RUN export IS_SNAPSHOT=false;                                                                      0.0s
 => ERROR [build 17/20] RUN dune build --release;                                                                           0.5s
------                                                                                                                           
 > [build 17/20] RUN dune build --release;:
0.411 /bin/sh: dune: not found
------
Dockerfile:48
--------------------
  46 |     RUN export IS_SNAPSHOT=false;
  47 |      
  48 | >>> RUN dune build --release;
  49 |     
  50 |     RUN dune install --relocatable \
--------------------
ERROR: failed to solve: process "/bin/sh -c dune build --release;" did not complete successfully: exit code: 127

which is strange. eval $(opam env); is ran but still dune: not found. However it's installed ok (into /root/.opam/liquidsoap/bin I presume). All the steps are followed as they are.

[vitoyucepi]

Dune operations should be combined into a single step with eval $(opam env).

These instructions don't work well as separate steps. You should combine them into a single step.
For example, export VAR will only be visible until the RUN ends. Another option is to use ENV or ARG.

[gAlleb]

Good to know, thank you.

[toots]

@vitoyucepi my experimentations with jemalloc have shown that we should actually not suggest building with. It adds a lot of memory overhead on normal run. I'll have more numbers in a future blog post.