Persistent Linux 'jails' on TrueNAS SCALE to install software (docker-compose, portainer, podman, etc.) with full access to all files via bind mounts.
USING THIS SCRIPT IS AT YOUR OWN RISK! IT COMES WITHOUT WARRANTY AND IS NOT SUPPORTED BY IXSYSTEMS.
TrueNAS SCALE can create persistent Linux 'jails' with systemd-nspawn. This script helps with the following:
- Setting up the jail so it won't be lost when you update SCALE
- Choosing a distro (Debian 12 strongly recommended, but Ubuntu, Arch Linux or Rocky Linux seem good choices too)
- Will create a ZFS Dataset for each jail if the
jailmakerdirectory is a dataset (easy snapshotting) - Optional: configuring the jail so you can run Docker inside it
- Optional: GPU passthrough (including nvidia GPU with the drivers bind mounted from the host)
- Starting the jail with your config applied
Despite what the word 'jail' implies, jailmaker's intended use case is to create one or more additional filesystems to run alongside SCALE with minimal isolation. By default the root user in the jail with uid 0 is mapped to the host's uid 0. This has obvious security implications. If this is not acceptable to you, you may lock down the jails by limiting capabilities and/or using user namespacing or use a VM instead.
Beginning with 24.04 (Dragonfish), TrueNAS SCALE includes the systemd-nspawn containerization program in the base system. Technically there's nothing to install. You only need the jlmkr.py script file in the right place. Instructions with screenshots are provided on the TrueNAS website. Start by creating a new dataset called jailmaker with the default settings (from TrueNAS web interface). Then login as the root user and download jlmkr.py.
cd /mnt/mypool/jailmaker
curl --location --remote-name https://raw.githubusercontent.com/Jip-Hop/jailmaker/main/jlmkr.py
chmod +x jlmkr.py
./jlmkr.py installThe jlmkr.py script (and the jails + config it creates) are now stored on the jailmaker dataset and will survive updates of TrueNAS SCALE. If the automatically created jails directory is also a ZFS dataset (which is true for new users), then the jlmkr.py script will automatically create a new dataset for every jail created. This allows you to snapshot individual jails. For legacy users (where the jails directory is not a dataset) each jail will be stored in a plain directory.
A symlink has been created so you can call jlmkr from anywhere (unless the boot pool is readonly, which is the default since SCALE 24.04). Additionally shell aliases have been setup, so you can still call jlmkr in an interactive shell (even if the symlink couldn't be created).
After an update of TrueNAS SCALE the symlink will be lost (but the shell aliases will remain). To restore the symlink, just run ./jlmkr.py install again or use the ./jlmkr.py startup command.
Creating jail with the default settings is as simple as:
jlmkr create myjailYou may also specify a path to a config template, for a quick and consistent jail creation process.
jlmkr create --config /path/to/config/template myjailOr you can override the default config by using flags. See jlmkr create --help for the available options. Anything passed after the jail name will be passed to systemd-nspawn when starting the jail. See the systemd-nspawn manual for available options, specifically Mount Options and Networking Options are frequently used.
jlmkr create --distro=ubuntu --release=jammy myjail --bind-ro=/mntIf you omit the jail name, the create process is interactive. You'll be presented with questions which guide you through the process.
jlmkr createAfter answering some questions you should have your first jail up and running!
# Call startup using the absolute path to jlmkr.py
# The jlmkr shell alias doesn't work in Init/Shutdown Scripts
/mnt/mypool/jailmaker/jlmkr.py startupIn order to start jails automatically after TrueNAS boots, run /mnt/mypool/jailmaker/jlmkr.py startup as Post Init Script with Type Command from the TrueNAS web interface. This creates the jlmkr symlink (if possible), as well as start all the jails with startup=1 in the config file.
jlmkr start myjailjlmkr listYou may want to execute a command inside a jail, for example from a shell script or a CRON job. The example below executes the env command inside the jail.
jlmkr exec myjail envThis example executes bash inside the jail with a command as additional argument.
jlmkr exec myjail bash -c 'echo test; echo $RANDOM;'jlmkr edit myjailOnce you've created a jail, it will exist in a directory inside the jails dir next to jlmkr.py. For example /mnt/mypool/jailmaker/jails/myjail if you've named your jail myjail. You may edit the jail configuration file, e.g. using the jlmkr edit myjail command (which uses the nano text editor). You'll have to stop the jail and start it again with jlmkr for these changes to take effect.
jlmkr remove myjailjlmkr stop myjailjlmkr restart myjailjlmkr shell myjailjlmkr status myjailjlmkr log myjailExpert users may use the following additional commands to manage jails directly: machinectl, systemd-nspawn, systemd-run, systemctl and journalctl. The jlmkr script uses these commands under the hood and implements a subset of their functions. If you use them directly you will bypass any safety checks or configuration done by jlmkr and not everything will work in the context of TrueNAS SCALE.
By default the jail will have full access to the host network. No further setup is required. You may download and install additional packages inside the jail. Note that some ports are already occupied by TrueNAS SCALE (e.g. 443 for the web interface), so your jail can't listen on these ports. This is inconvenient if you want to host some services (e.g. traefik) inside the jail. To workaround this issue when using host networking, you may disable DHCP and add several static IP addresses (Aliases) through the TrueNAS web interface. If you setup the TrueNAS web interface to only listen on one of these IP addresses, the ports on the remaining IP addresses remain available for the jail to listen on.
See Advanced Networking for more.
Using the docker config template is recommended if you want to run docker inside the jail. You may of course manually install docker inside a jail. But keep in mind that you need to add --system-call-filter='add_key keyctl bpf' (or disable seccomp filtering). It is not recommended to use host networking for a jail in which you run docker. Docker needs to manage iptables rules, which it can safely do in its own networking namespace (when using bridge or macvlan networking for the jail).
Additional documentation contributed by the community can be found in the docs directory.
TODO: write comparison between systemd-nspawn (without jailmaker), LXC, VMs, Docker (on the host).
The rootfs image jlmkr.py downloads comes from the Linux Containers Image server. These images are made for LXC. We can use them with systemd-nspawn too, although not all of them work properly. For example, the alpine image doesn't work well. If you stick with common systemd based distros (Debian, Ubuntu, Arch Linux...) you should be fine.
To visually distinguish between a root shell inside the jail and a root shell outside the jail, it's possible to colorize the shell prompt. When using a debian jail with the bash shell, you may run the following command to get a yellow prompt inside the jail (will be activated the next time you run jlmkr shell myjail):
echo "PS1='${debian_chroot:+($debian_chroot)}\[\033[01;33m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '" >> ~/.bashrc