Skip to content

Build from Source (Developer Workflow)

Jump to bottom
Alex Reustle edited this page Feb 4, 2021 · 10 revisions

Developing without conda-build

Normally to build the tools for release a developer would follow the conda-build procedures to automatically handle all that follows. However this method is redundant, slow and poorly suited for an iterative development and debugging style. Consequently we break the procedure into its component parts and enable each of them to be performed independently.

Automated scripts for performing these builds are available at https://github.com/fermi-lat/fermi_dev_scripts

Development Requirements

The Fermitools is a compiled conda package written in C++ and python for MacOSX and Linux operating systems. To build the tools you will need:

  • A Linux or MacOSX based system
  • An anaconda distribution (Miniconda 2 or 3 will suffice)
  • fermi-repoman: conda install -c fermi fermi-repoman
  • If using MacOSX you will need to install Xcode, or acquire the OSX SDK.

Quick build example

Here is a quick overview example of the steps involved in preparing, downloading and building the tools from source. This example applies to linux systems, there is an additional complication for macosx systems described below.

  1. Prepare your anaconda build environment, install fermi-repoman and download the fermi_dev_scripts
conda install -c fermi fermi-repoman
git clone https://github.com/fermi-lat/fermi_dev_scripts.git
  1. Create a conda environment populated with necessary dependencies
export FERMI_REF=Fermitools-2.0.0
export FERMI_CONDA_ENV=my_fermi_env
./fermi_dev_scripts/conda_fermi_build_deps_explicit.sh
conda activate my_fermi_env
  1. Download the source code and build the tools
export CPU_COUNT=4
source fermi_dev_scripts/build.sh

Mac build complications

Building from source on MacOSX requires additional steps. First, MacOSX builds require an Xcode SDK file., commonly this can be acquired by installing the latest version of XCode. Additionally you must point the build to this SDK by setting your CONDA_BUILD_SYSROOT environment variable. On most commonly configured systems the XCode SDKs are installed at /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk

Also required is the MACOSX_DEPLOYMENT_TARGET environment variable, of which only 10.9 is supported.

If left unset these environment variables will be defaulted to the values below.

CONDA_BUILD_SYSROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk  
MACOSX_DEPLOYMENT_TARGET=10.9

In depth description of source build options.

Conda Build-time Externals

Before building the tools the user must create a special conda environment for development and install into that environment all such external packages as are dependencies for the fermitools. These "Build dependencies" or "Build externals" can be acquired in various ways, but the most straightforward is to copy them explicitly from a successful past build of the tools. The exact mechanism for which has been coded into a suite of scripts available at fermi_dev_scripts.

FERMI_REF

In git a ref is a commit SHA, branch name, or git tag. Our system is designed to share branches and tags across multiple repositories. Each release and official beta of the Fermitools is automatically assigned a git tag in the fermi-lat github organization by our automated build pipeline. These tags follow the format Fermitools-X.X.X. Choose a common Tag or branch name and use that as your base for building dependencies. The list of generated tagnames can be found here

Set needed Environment Variables

The development scripts read variables from the user's environment. These may be customized for your target REF, environment and hardware, etc.

# Linux
export FERMI_REF=Fermitools-2.0.0
export FERMI_CONDA_ENV=my_fermi_env
export CPU_COUNT=4

# MacOSX
export FERMI_REF=Fermitools-2.0.0
export FERMI_CONDA_ENV=my_fermi_env
export CPU_COUNT=4
export CONDA_BUILD_SYSROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk  
export MACOSX_DEPLOYMENT_TARGET=10.9

Create a conda environment populated with build-time dependencies (Explicit).

Create a conda environment from the exact dependency tarballs used to create the original build environment.

fermi_dev_scripts/conda_fermi_deps_explicit.sh
conda activate my_fermi_env

The scripts avail themselves of the conda package manager's --explicit flag to download tarballs from the URLs stored in a text file. This file is located in the fermi-lat github organization under one of 2 repositories: Fermitools-explicit-build-deps-linux Fermitools-explicit-build-deps-macosx with a git tag indicating which release it was compiled for.

The files and tags are uploaded automatically by the Azure Pipelines automated build pipeline.

Obtain Sources and Build the Tools

Sources for the fermitools are spread across multiple repositories in the fermi-lat github organization. In order to obtain them we have a repoman stage of the build script that optionally checks out the sources with a branch, tag, or commit hash taken from the $FERMI_REF environment variable. The exact list of packages to checkout is stored in Fermitools-conda under the packageList.txt file.

Optional Repoman checkout and scons build

source fermi_dev_scripts/build.sh

Disable Repoman checkout during the build

export FERMI_NO_CHECKOUT=true
source fermi_dev_scripts/build.sh

Disable SCONS Install step during the build

export FERMI_NO_INSTALL=true
source fermi_dev_scripts/build.sh
Clone this wiki locally