new configuration for using nf-core pipelines at University of Luebeck #592
Conversation
|
|
||
| ```bash | ||
| module load nextflow | ||
| module load singularity |
There was a problem hiding this comment.
Are loaded modules carried through to jobs or should this be in a process.beforeScript as well?
There was a problem hiding this comment.
The modules are carried through. It's not needed in a process.beforeScript
docs/uzl_omics.md
Outdated
|
|
||
| nf-core requires a Nextflow version 22.10.1 or higher, so you have to intall a more recent version first. | ||
|
|
||
| For Nextflow versions newer than 23.07.0-edge, it is necessary to explicitly mount the home directory using the command: |
There was a problem hiding this comment.
There seems to be some buggy modules not being happy without a writable $HOME, is this because of this or some other thing?
There was a problem hiding this comment.
Without it causes an error at check_samplecheet.py as can seen here
https://nfcore.slack.com/archives/CE8SSJV3N/p1698833301245139
The nextflow version didn't bind the home directory, so there was no access to $HOME/.nextflow/assets/nf-core/rnaseq/bin
It only worked with providing NXF_SINGULARITY_HOME_MOUNT=true
There was a problem hiding this comment.
It seems I made the last comments there ;)
Was the outcome that the singularity setup was intentionally broken (not allowing mount controls)? I guess that would limit to $HOME if that was the case, but then again, I'd definitely try raising it for a chance to have it fixed first.
There was a problem hiding this comment.
Could you please explain what you mean and how we can get it fixed?
The singularity is installed by the HPC maintainers, and we are currently unsure why this problem occurs when we do not explicitly ask to mount $HOME using the global variable NXF_SINGULARITY_HOME_MOUNT.
In our understanding, it is rather a Nextflow related issue, because previous versions (e.g. 22.10.1) of Nextflow worked fine without this variable decoration. Starting from version 23.07.0-edge, nextflow documentation says that the default is set to false when it comes to mounting the home directory.
There was a problem hiding this comment.
As was mentioned in the slack thread, it is very likely a good idea to reach out to your systems people and let them now that their singularity configuration (disallowing user binds) makes things more difficulty for you.
I obviously don't know how your systems are set up exactly, and maybe everything is under users' home directories - then making sure nextflow doesn't disable home binds would be good enough (although the user experience is lousy as it currently requires setting that environment variable). But obviously that doesn't cover anything run outside of the home directories.
A setup that doesn't have singularity stopping if a bind is provided could use singularity.autoMounts to pass the binds nextflow knows of that would be required (although in some cases with links going back and forth, it might be a good idea to help with specifying some binds explicitly).
So, to sum up, I'd say the singularity configuration is weird if it doesn't allow the user specificying bind mounts (which it seemed from the slack thread), and I'd generally suggest trying to fix that since it breaks a lot of use cases.
There was a problem hiding this comment.
We will work on fixing this issue. For now I have edited the documentation file and erased this part, to not cause any conflicts.
|
Looks much better, but I made a few new comments on the documentation. Also, the linter still complains on the documentation page, can you please update? (Can e.g. be done with running |
docs/uzl_omics.md
Outdated
|
|
||
| nf-core requires a Nextflow version 22.10.1 or higher, so you have to intall a more recent version first. | ||
|
|
||
| For Nextflow versions newer than 23.07.0-edge, it is necessary to explicitly mount the home directory using the command: |
There was a problem hiding this comment.
We will work on fixing this issue. For now I have edited the documentation file and erased this part, to not cause any conflicts.
Adding a configuration file for the OMICS cluster of the University of Luebeck.
Adding the documentation file uzl_omics.md for the OMICS cluster of the University of Luebeck.
adding custom profile for the University of Luebeck
Adding custom profile for the OMICS cluster of the University of Luebeck
Adding profile name for the OMICS Cluster of the University of Luebeck.
fix nextflow version for home mount in uzi_omics.md
Erasing the unnecessary configuration settings.
For the UzL OMICS cluster it is necessary to be in $WORK directory. After erasing the workDir in the configuration file, it is even more necessary to provide it in the batch-script.
fixed syntax with prettier
Adding the non-mandatory information. After fixing syntax with prettier.
Fix according to the suggestions. Erased that running the pipeline in `$WORK` is necessary.
erasing the part with NXF_SINGULARITY_HOME_MOUNT that causes conflicts
corrected the syntax using prettier
name: New Config
about: A new cluster config
Please follow these steps before submitting your PR:
[WIP]in its titlemasterbranchSteps for adding a new config profile:
conf/directorydocs/directorynfcore_custom.configfile in the top-level directoryREADME.mdfile in the top-level directoryprofile:scope in.github/workflows/main.yml