Env. variables SINGULARITY_TEMPDIR, SINGUALRITY_CACHEDIR, TMPDIR not passed to singularity

[natalie-perlin]

Version of Singularity
What version of Singularity are you using? Run:

$ singularity --version
singularity version 3.8.6-1.el7

Describe the bug
A clear and concise description of what the bug is.
Attempting to build a singularity image from the docker container. Getting errors "no space left on device" for the /tmp directory, even though the environment variables SINGULARITY_TEMPDIR, SINGULARITY_TEMPDIR, and TMPDIR are set to point to a different location, as following, before running singularity command:

export SINGULARITY_TEMPDIR=/lustre/tmp
export SINGUALRITY_CACHEDIR=/lustre/cache
export TMPDIR=/lustre/tmp

Attempted to run with both regular user and with sudo, with the same outcome.
sudo -E singularity shell -B /lustre docker://noaaepic/ubuntu20.04-intel-landda:develop
sudo -E singularity shell -B /lustre docker://noaaepic/ubuntu20.04-intel-landda:develop

It fails on systems with smaller /tmp space, producing error such as:

FATAL: Unable to handle docker://noaaepic/ubuntu20.04-intel-landda:develop uri: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack layer: unpack entry: usr/local/lib/python3.8/dist-packages/numpy.libs/libopenblas64_p-r0-15028c96.3.21.so: unpack to regular file: short write: write /tmp/build-temp-956582340/rootfs/usr/local/lib/python3.8/dist-packages/numpy.libs/libopenblas64_p-r0-15028c96.3.21.so: no space left on device

To Reproduce
Steps to reproduce the behavior:

I performed a simple test to verify that the env. variables $SINGULARITY_TEMPDIR and $SINGULARITY_CACHEDIR are not passed into the singularity, by shelling into a smaller docker image. Same behavior occurs when the sandbox is created first.

export SINGULARITY_TEMPDIR=/lustre/tmp
export SINGUALRITY_CACHEDIR=/lustre/cache
export TMPDIR=/lustre/tmp
# verify:
env | grep SINGULARITY
(prompt:)
SINGULARITY_TEMPDIR=/lustre/tmp
SINGULARITY_CACHEDIR=/lustre/cache

sudo -E singularity shell -B /lustre docker://ubuntu:latest
(after it's finished building, query the env. variables:)

Singularity> env | grep SINGULARITY
SINGULARITY_NAME=9a0bdde4188b896a372804be2384015e90e3f84906b750c1a53539b585fbbe7f
SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh
SINGULARITY_CONTAINER=/lustre/cache/cache/oci-tmp/9a0bdde4188b896a372804be2384015e90e3f84906b750c1a53539b585fbbe7f
SINGULARITY_BIND=/lustre
SINGULARITY_COMMAND=shell
Singularity> env | grep TMPDIR
Singularity> echo $TMPDIR

Singularity>

(UPD:) When it is launched as a regular user, the TMPDIR variable does happen to be passed to singularity
singularity shell -B /lustre docker://ubuntu:latest

Singularity> echo $TMPDIR
/lustre/tmp
Singularity>

It demonstrates that the singularity does not receive the variables set for cache and tmpdir.

Expected behavior

Expecting to see $SINGUALRITY_CACHEDIR, $SINGUALRITY_TEMPDIR, and $TMPDIR variables being passed when singularity is launched, and appear when prompted in the singularity shell, e.g.:

Singularity> echo $SINGULARITY_TEMPDIR
/lustre/tmp
Singularity> echo $SINGULARITY_CACHEDIR
/lustre/cache
Singularity> echo $TMPDIR
/lustre/tmp

OS / Linux Distribution
Which Linux distribution are you using?

Linux 3.10.0-1160.59.1.el7.x86_64

$ cat /etc/os-release

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Installation Method
Write here how you installed SingularityCE. Eg. RPM, source.

(not sure, not installed myself)

Additional context
Anything else which might be relevant. E.g. if the bug only occurs on a specific filesystem, or kernel version etc.
The bug does not occur on the system when the /tmp is sufficiently large.

[dtrudg]

$ singularity --version
singularity version 3.8.6-1.el7

Hello - this is not a release of SingularityCE (this project).

Singularity forked into what is now known as Apptainer, and SingularityCE, during the 3.8 series. The last 3.8 release on this (SingularityCE) side of the fork was 3.8.4. Tha apptainer side of the fork made some additional 3.8.x releases before renaming to apptainer, and your 3.8.6-1 installation is one of those.

Any version of singularity reporting itself as 3.8.6 comes from the retired repository on the apptainer side of the fork:

https://github.com/apptainer/singularity

If you would like us to investigate this issue under SingularityCE, please could you replicate it using a current supported version of SingularityCE? The latest version is 3.11.0 - https://github.com/sylabs/singularity/releases/tag/v3.11.0

In EPEL, SingularityCE 3.10.5 is currently available via yum install singularity-ce.

Thank you.

[dtrudg]

Regarding the report you have made above, I don't believe there is a bug here.

You are export -ing SINGULARITY_TEMPDIR, but checking for TMPDIR - it appears correct as a normal user, because TMPDIR is already set in your environment.

You need to export SINGULARITY_TMPDIR (no E).

See: https://docs.sylabs.io/guides/3.11/user-guide/build_env.html#temporary-folders

[dtrudg]

I believe you might be using ufs-srweather-app? I have written a PR correcting their guide here:

ufs-community/ufs-srweather-app#622

[natalie-perlin]

@dtrudg - thank you a lot for your comment! Yes, it is very helpful, indeed the "SINGULARITY_TMPDIR" instead of "...TEMPDIR" is the culprit.
The only question remains, however, why these two variables, "SINGULARITY_TMPDIR" and "SINGULARITY_CACHEDIR" do not appear when queried inside running instance of singularity, e.g.:

Singularity> env | grep SINGULARITY

It would be really helpful to have these env. variables shown when queried, for debugging or other purposes, such as binding correct directories to allow for "...TMPDIR" and "...CACHEDIR" creation.

Thank you!

[dtrudg]

@natalie-perlin - we don't pass these variables into the container, because if we did they would affect the behavior if another 'singularity' was run inside the container. Nested execution of containers is more common then might be expected.

To verify the effect of SINGULARITY_ variables, you can usually look at the singularity --debug ... ouput.

E.g.

$ export SINGULARITY_TMPDIR=/lustre/tmp

$ singularity -d build test.sif docker://alpine
DEBUG   [U=1001,P=1527627] persistentPreRun()            Singularity version: 3.11.0
DEBUG   [U=1001,P=1527627] persistentPreRun()            Parsing configuration file /usr/local/etc/singularity/singularity.conf
DEBUG   [U=1001,P=1527627] handleConfDir()               /home/dtrudg-sylabs/.singularity already exists. Not creating.
DEBUG   [U=1001,P=1527627] handleRemoteConf()            Ensuring file permission of 0600 on /home/dtrudg-sylabs/.singularity/remote.yaml
DEBUG   [U=1001,P=1527627] getCacheParentDir()           environment variable SINGULARITY_CACHEDIR not set, using default image cache
DEBUG   [U=1001,P=1527627] newBundle()                   Created temporary directory "/lustre/tmp/bundle-temp-4276329060" for the bundle
DEBUG   [U=1001,P=1527627] newBundle()                   Created directory "/lustre/tmp/build-temp-3684074322/rootfs" for the bundle
...

Note how the line there shows that we Created temporary directory in the /lustre/tmp location, and there is a line that's talking about SINGULARITY_CACHEDIR also.

These environment variables are read from the host, not the container, so you can also always check if they are set by running env or sudo -E env.