Building binaries on FreeBSD

FreeBSD binaries are built using Poudriere, a FreeBSD-specific tool to build binary packages using a clean-room environment. Poudriere:

• Creates and manages lightweight virtual environments (jails) for i386 and amd64
• Keeps virtual environments up to date
• Manages the building process, logging and packaging. It can expose results via the HTTP interface.

Another tool being used is portshaker which helps to merge different FreeBSD port trees and keep them up to date. A FreeBSD port tree describes how applications should be built into binary packages and installed. This is the standard way of building third-party software on FreeBSD.

Custom ports tree

Since node-sass and many npm packages are not yet part of the official FreeBSD ports tree, we maintain a custom ports tree which contains the necessary elements. That's why we need portshaker to combine the official tree (containing python, compilers, node) and the custom tree.

We currently have the following custom ports:

Port name	Description
www/npm	npm port modified to support workaround segfault in i386 environment
textproc/libsass	updated libsass FreeBSD port
textproc/sassc	sassc to match the updated libsass
textproc/node-sass	Instructions how to build node-sass (depends on the libsass port)

Because www/npm already exists in the official FreeBSD tree we need to have a higher version number in our custom port, which is easily achieved by bumping up PORTREVISION. If we have an older npm version, portshaker complains:

www/npm: port version going backward (I will not merge this port)!

Challenges to using FreeBSD ports infrastructure

FreeBSD port building is split into certain well-defined phases, like fetch, depend, extract, building and stage and install. Every phase has its own restrictions - for example during extract/build and later we are not supposed to have network access. Ports working directories can be written during extraction phase and later. This is a much more strict model than that one used by the npm, where fetching of dependencies is mixed with the construction of the package. To work this around we try to fetch all dependencies in advance into a download directory that serves also as the npm package cache. NPM_CONFIG_CACHE is set to this directory (usually /usr/ports/distfiles/npm) to achieve this.

We supply node headers and gypi files installed by the FreeBSD node ports to node-gyp for perusal.

Since poudriere enforces the above restrictions (network access, etc.) during build the building machine fails at the npm install stage when trying to resolve DNS hostname. If all dependencies are provided in advance, npm proceeds with the installation using locally cached files and does not try to access the network.

Environments available

We can build binaries for FreeBSD 9.3 and 10.1. For official binaries we ship only 10.1 binaries because 9.3 ships only with gcc 4.2.1 by default and we need newer C++ runtime (stdlibc++ library). The packages for 9.3 are build and tested; but putting them into node-sass binary distribution is currently not possible; see sass/node-sass#733 for more details. Using node-sass is perfectly possible on those platforms if the newer gcc is installed.

FreeBSD version	architecture	Environment name
10.1-RELEASE-p17	amd64	iojs_10_1_amd64
10.1-RELEASE-p17	amd64	nodejs010_10_1_amd64
10.1-RELEASE-p17	amd64	nodejs_10_1_amd64
10.1-RELEASE-p17	i386	iojs_10_1_i386
10.1-RELEASE-p17	i386	nodejs010_10_1_i386
10.1-RELEASE-p17	i386	nodejs_10_1_i386
10.1-RELEASE-p17	amd64	iojs42_10_1_amd64
10.1-RELEASE-p17	i386	iojs42_10_1_i386
10.1-RELEASE-p17	amd64	iojs44_10_1_amd64
10.1-RELEASE-p17	i386	iojs44_10_1_i386
10.1-RELEASE-p17	amd64	iojs45_10_1_amd64
10.1-RELEASE-p17	i386	iojs45_10_1_i386
10.1-RELEASE-p19	amd64	node46_10_1_amd64
10.1-RELEASE-p19	i386	node46_10_1_i386
9.3-RELEASE-p22	amd64	iojs_9_3_amd64
9.3-RELEASE-p22	amd64	nodejs010_9_3_amd64
9.3-RELEASE-p22	amd64	nodejs_9_3_amd64
9.3-RELEASE-p22	i386	iojs_9_3_i386
9.3-RELEASE-p22	i386	nodejs010_9_3_i386
9.3-RELEASE-p22	i386	nodejs_9_3_i386
9.3-RELEASE-p22	amd64	iojs42_9_3_amd64
9.3-RELEASE-p22	i386	iojs42_9_3_i386
9.3-RELEASE-p22	amd64	iojs44_9_3_amd64
9.3-RELEASE-p22	i386	iojs44_9_3_i386
9.3-RELEASE-p22	amd64	iojs45_9_3_amd64
9.3-RELEASE-p22	i386	iojs45_9_3_i386
9.3-RELEASE-p25	amd64	node46_9_3_amd64
9.3-RELEASE-p25	i386	node46_9_3_i386

How to update and maintain the custom node-sass port

• Update libsass and sassc ports. Test them on different architectures. Usually trying to run sassc in the virtual environment is enough to check if all needed C++ libraries are in place.

• Check if www/npm port has been updated in the main ports tree. Update custom npm accordingly if this happened.

• To bootstrap the process we provide a stub package.json file to pre-download all the dependencies in advance. This file must contain all dependencies and the current version, but must not contain node-sass custom build scripts.

• Push the release version into the FreeBSD git tree. Note the last commit identifier.

• Delete old distfiles by running make distclean and delete the distinfo file.

• Update the Makefile to reflect the new version and the commit identifier.

• Run make makesum to fetch the new .tar.gz file. GitHub sometimes has trouble to create a .tar.gz distribution right after pushing the new commit into the branch. distinfo is recreated at this point.

• Build the port without using poudriere by running make stage

• Update the pkg-plist by running find:

( cd `make -V STAGEDIR`/usr/local && find bin lib -type f -o -type l ) | sort | grep -v vendor > pkg-plist

• Add @dir lib/node_modules/node-sass/src/libsass and @dir lib/node_modules/node-sass/test/fixtures/spec entries at the end of the pkg-plist file.

• Update version number in the nodesass-repack.sh script.

Building process

1. A shell script is used to start poudriere run for i386 and amd64 to build FreeBSD binaries for 10.1-RELEASE.

2. After the packages are built and placed into /usr/local/poudriere/data/packages/iojs42_10_1_i386-exp/All/node-sass-3.0.0.b7.txz (and friends) we are using a nodesass-repack.sh script to extract the binaries only and put them into a checked out directory of https://github.com/sass/node-sass-binaries repository.

Sample poudriere build log

An example for a successful poudriere output for io.js 1.0.4 32-bit:

[00:00:01] ====>> Creating the reference jail... done
[00:01:06] ====>> Mounting system devices for iojs42_10_1_i386-exp
[00:01:06] ====>> Mounting ports/packages/distfiles
[00:01:06] ====>> Stashing existing package repository
[00:01:07] ====>> Mounting ccache from: /usr/obj/cache
[00:01:07] ====>> Mounting packages from: /usr/local/poudriere/data/packages/iojs42_10_1_i386-exp
[00:01:07] ====>> Mounting /var/db/ports from: /usr/local/etc/poudriere.d/iojs42_10_1_i386-options
[00:01:07] ====>> Appending to make.conf: /usr/local/etc/poudriere.d/make.conf
[00:01:07] ====>> Appending to make.conf: /usr/local/etc/poudriere.d/exp-make.conf
[00:01:07] ====>> Appending to make.conf: /usr/local/etc/poudriere.d/iojs42_10_1_i386-make.conf
/etc/resolv.conf -> /usr/local/poudriere/data/.m/iojs42_10_1_i386-exp/ref/etc/resolv.conf
[00:01:07] ====>> Starting jail iojs42_10_1_i386-exp
[00:01:07] ====>> Logs: /usr/local/poudriere/data/logs/bulk/iojs42_10_1_i386-exp/2015-04-22_10h02m02s
[00:01:07] ====>> Loading MOVED
[00:01:07] ====>> Calculating ports order and dependencies
[00:01:13] ====>> Sanity checking the repository
[00:01:13] ====>> Checking packages for incremental rebuild needed
[00:01:15] ====>> Deleting stale symlinks
[00:01:15] ====>> Deleting empty directories
[00:01:15] ====>> Cleaning the build queue
[00:01:16] ====>> Recording filesystem state for prepkg... done
[00:01:17] ====>> Building 1 packages using 1 builders
[00:01:17] ====>> Starting/Cloning builders
[00:01:49] ====>> Hit CTRL+t at any time to see build progress and stats
[00:01:50] ====>> [01][00:00:00] Starting build of textproc/node-sass
[00:03:48] ====>> [01][00:01:58] Finished build of textproc/node-sass: Success
[00:03:50] ====>> Stopping 1 builders
iojs42_10_1_i386-exp-job-01: removed
iojs42_10_1_i386-exp-job-01-n: removed
[00:03:58] ====>> Creating pkgng repository
Creating repository in /tmp/packages: 100%
Packing files for repository: 100%
[00:03:59] ====>> Committing packages to repository
[00:03:59] ====>> Removing old packages
[00:03:59] ====>> Built ports: textproc/node-sass
[iojs42_10_1_i386-exp] [2015-04-22_10h02m02s] [committing:] Queued: 1  Built: 1  Failed: 0  Skipped: 0  Ignored: 0  Tobuild: 0   Time: 00:02:52
[00:03:59] ====>> Logs: /usr/local/poudriere/data/logs/bulk/iojs42_10_1_i386-exp/2015-04-22_10h02m02s
[00:03:59] ====>> Cleaning up
iojs42_10_1_i386-exp: removed
iojs42_10_1_i386-exp-n: removed
[00:04:00] ====>> Umounting file systems