Singularity uses python libraries outside container

[grst]

Bug report

Expected behavior and actual behavior

Expected: The process fails, as Biopython is not available in a vanilla python container
Actual: The process succeeds, as the .local directory in my home directory gets mounted into the container

This is a problem as pipelines may not be reproducible anymore as a different version of a Python package than
installed in the container may be used. This may happen with R packages, too.

Steps to reproduce the problem

nextflow.config

singularity.enabled = true
singularity.autoMounts = true

singularity-nohome.nf

#!/usr/bin/env nextflow

nextflow.enable.dsl = 2

process test_singularity {

    container = "https://depot.galaxyproject.org/singularity/python%3A3.9"

    script:
    """
    #!/usr/bin/env python

    import Bio 
    """
}

workflow {
    test_singularity()
}

NB: to reproduce, the python version of the container must match the python version on the system running nextflow

Command executed:

pip install --use Biopython
./singularity-nohome.nf -c nextflow.config

Program output

(Copy and paste here output produced by the failing execution. Please highlight it as a code block. Whenever possible upload the .nextflow.log file.)

Environment

• Nextflow version: version 21.04.0 build 5552
• Java version: openjdk version "11.0.8-internal" 2020-07-14
• Operating system: Linux
• Bash version: GNU bash, version 5.1.8(1)-release (x86_64-pc-linux-gnu)
• Singularity version: singularity version 3.8.3

Proposed solution

Always call singularity with the --no-home option.

---

CC @riederd

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[grst]

bump, as this is very bad for reproducibility. Linking the related nf-core discussion: https://nfcore.slack.com/archives/CE5LG7WMB/p1643152746064800

…

On Thu, 7 Apr 2022 at 01:49, stale[bot] ***@***.***> wrote: This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. — Reply to this email directly, view it on GitHub <#2348 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVZRV7PS5YY5W6GUIVEW33VDYPITANCNFSM5E2HHHCA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Grelot]

I got the same problem with Rscript. Rscript is located at /usr/bin/ in both my computer and the container. Nextflow seems to load the Rscript from my computer instead of the container. How to solve it ?

[grst]

This is the workaround used in nf-core:
https://github.com/nf-core/rnaseq/blob/89bf536ce4faa98b4d50a8ec0a0343780bc62e0a/nextflow.config#L200-L205

I really think this should be fixed on the nexflow side though!

[Grelot]

I tried to run my nextflow with and without this fix in my nextflow.config, unfortunally it changes nothing to the behaviour.
I also tried to run nextflow with --no-home option for singularity or by setting singularity.autoMounts = false

It doesn't change anything. This is strange because on a non-DSL2 nextflow, the singularity container is working well with Rscript.
And in my DSL2 nextflow, singularity is working well for dependencies that are not R/python/...

[grst]

how do you pass the --no-home flag?
This has worked well for me before:
https://github.com/icbi-lab/luca/blob/6f2ea7203272b6d6f7e415f7abf0d247c8368ed4/nextflow.config#L11-L12

Setting autoMounts = false has no effect on the home directory being mounted.

[Grelot]

It fixed my problem ! Indeed I use it only on the concerned modules in conf/modules.config. Thank you a lot.
However, this remains a temporary fix as I will not be able to use Rscript or python script with external files as inputs.

PS: I used the wrong notation:

singularity {
    runOptions = "--no-home --clean-env"
}

instead of

process {
    withName: 'myprocess' {
        container = '/path/to/my/container'
        containerOptions = "--no-home"
    }

}

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[grst]

I still think this should be fixed.

[pditommaso]

is it not containerOptions = "--no-home" a solution for this?

[riederd]

shouldn't be this the default?

[pditommaso]

Could not break the execution of pipeline that are relying on that?

[riederd]

yes it could, but just as it breaks pipelines that rely on using libraries/tools/software-versions that are packaged into the singularity image.

I think the question is, what is more important then.

For me, as a pipeline developer, it would be more important that the pipeline is reliably using the software (versions) that I pack into the image, so things remain reproducible.

[grst]

Well, you never know, but

• the whole point of containers is to separate the software used by a process from the software available on a system
• workflows relying on the home directly being mounted in a container are likely doing something wrong (i.e. files not staged into the work dir by nextflow).

Error messages arising due to wrong package versions being used are usually obscure and hard to debug. If a file is not found, it's pretty obvious.

[pditommaso]

Fair enough. This can be added provided there's an option to opt-out e.g. NXF_ENABLE_SINGULARITY_HOME_MOUNT.

I leave this as a community effort, feel free to provide a PR to address it.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[GallVp]

If someone comes looking for why some of their JAVA based programs are failing after the addition of --no-home, following java options may help.

-Djava.util.prefs.userRoot=user_prefs -Duser.home=user_home