Use of OCI Containers

[edmundmiller]

Branching off from #4080 (comment)

Documenting a chat with @drpatelh

The issue used to be that the conversion from docker container to sif would create temporary cache files in a users home directory, and that was a user pain point on HPC systems.

I think there's two things that might fix that:

1. The singularity cache is written to the work directory. Not sure if the conversion files are in there or not.
2. The OCI containers thing might fix it? What if users don't have a new enough singularity version?

Essentially the goal is to get rid of the scary container logic and just point to one container.

    conda "${moduleDir}/environment.yml"
+   container 'nf-core/modules/chromap:chromap--a6833a982c474692
-   container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
-       'https://depot.galaxyproject.org/singularity/chromap:0.2.4--hd03093a_0' :
-       'biocontainers/chromap:0.2.5--hd03093a_0' }"

[marcodelapierre]

Hi @emiller88 @drpatelh , as I was suggesting here : nextflow-io/nextflow#4543

Adding the following should make it: singularity.runOptions = '--env PATH=/opt/conda/bin:\\\$PATH'

(if you move to Wave, you won't need this any more)

Feel free to let me know how it goes.

[marcodelapierre]

I think at this stage the OCI requirement may not be the most appropriate wording. There are more general ways to overcome the issue.

[edmundmiller]

(if you move to Wave, you won't need this any more)

This is blocked by seqeralabs/wave#323.

[edmundmiller]

From @ewels:

• Takes quite a lot of time / CPU and additional disk space (all of the docker layers are cached, singularity tmp dirs fill up)
• Also you need Singularity installed, for nf-core download it's nice if you can just download a file with pure python
• As often people are downloading on their laptops (where installing Singularity is not needed + a hassle) to transfer to a HPC

[marcodelapierre]

Update: going to improve Nextflow functionalities with Singularity with this PR.

• Singularity starting with version 3.0, and any Apptainer version, both can take care of image caching. By default done in ~/.singularity, but can be customised with SINGULARITY_CACHEDIR/APPTAINER_CACHEDIR env variable; this will likely be enabled in NF by singularity.ociAuto = true . This is probably the key improvement in this context
• Newest native OCI in Singularity 4 does not significantly improve things

Referring to the HPC issue with cache filling up:

• ociAuto above will disable NF saving an image file in the pipeline work directory, reducing duplication of image files
• container image blobs and images will be only saved by Singularity in its cache dir (this cannot be prevented)
• what can further be improved at the Singularity level is setting the cache dir to a location with more convenient disk quotas in an HPC cluster

Please keep an eye on nextflow-io/nextflow#4548 ; happy to take it from there from the remaining issues after that is merged.

[marcodelapierre]

Update: going to improve Nextflow functionalities with Singularity with this PR.

• Singularity starting with version 3.0, and any Apptainer version, both can take care of image caching. By default done in ~/.singularity, but can be customised with SINGULARITY_CACHEDIR/APPTAINER_CACHEDIR env variable; this will likely be enabled in NF by singularity.ociAuto = true . This is probably the key improvement in this context
• Newest native OCI in Singularity 4 does not significantly improve things

Keep an eye on the PR, once merged I am happy to take it from there with you Edmund, on the remaining issues you are facing.