Update module and pipeline template #1336

drpatelh · 2021-11-25T17:08:10Z

…release

JoseEspinosa

Awesome job @drpatelh 🚀
Tests are failing but I think we can merge and then work on them!

JoseEspinosa · 2021-11-25T18:39:54Z

nf_core/pipeline-template/docs/usage.md

-### Tool-specific options
-
-For the ultimate flexibility, we have implemented and are using Nextflow DSL2 modules in a way where it is possible for both developers and users to change tool-specific command-line arguments (e.g. providing an additional command-line argument to the `STAR_ALIGN` process) as well as publishing options (e.g. saving files produced by the `STAR_ALIGN` process that aren't saved by default by the pipeline). In the majority of instances, as a user you won't have to change the default options set by the pipeline developer(s), however, there may be edge cases where creating a simple custom config file can improve the behaviour of the pipeline if for example it is failing due to a weird error that requires setting a tool-specific parameter to deal with smaller / larger genomes.
-
-The command-line arguments passed to STAR in the `STAR_ALIGN` module are a combination of:
-
-* Mandatory arguments or those that need to be evaluated within the scope of the module, as supplied in the [`script`](https://github.com/nf-core/rnaseq/blob/4c27ef5610c87db00c3c5a3eed10b1d161abf575/modules/nf-core/software/star/align/main.nf#L49-L55) section of the module file.
-
-* An [`options.args`](https://github.com/nf-core/rnaseq/blob/4c27ef5610c87db00c3c5a3eed10b1d161abf575/modules/nf-core/software/star/align/main.nf#L56) string of non-mandatory parameters that is set to be empty by default in the module but can be overwritten when including the module in the sub-workflow / workflow context via the `addParams` Nextflow option.
-
-The nf-core/rnaseq pipeline has a sub-workflow (see [terminology](https://github.com/nf-core/modules#terminology)) specifically to align reads with STAR and to sort, index and generate some basic stats on the resulting BAM files using SAMtools. At the top of this file we import the `STAR_ALIGN` module via the Nextflow [`include`](https://github.com/nf-core/rnaseq/blob/4c27ef5610c87db00c3c5a3eed10b1d161abf575/subworkflows/nf-core/align_star.nf#L10) keyword and by default the options passed to the module via the `addParams` option are set as an empty Groovy map [here](https://github.com/nf-core/rnaseq/blob/4c27ef5610c87db00c3c5a3eed10b1d161abf575/subworkflows/nf-core/align_star.nf#L5); this in turn means `options.args` will be set to empty by default in the module file too. This is an intentional design choice and allows us to implement well-written sub-workflows composed of a chain of tools that by default run with the bare minimum parameter set for any given tool in order to make it much easier to share across pipelines and to provide the flexibility for users and developers to customise any non-mandatory arguments.
-
-When including the sub-workflow above in the main pipeline workflow we use the same `include` statement, however, we now have the ability to overwrite options for each of the tools in the sub-workflow including the [`align_options`](https://github.com/nf-core/rnaseq/blob/4c27ef5610c87db00c3c5a3eed10b1d161abf575/workflows/rnaseq.nf#L225) variable that will be used specifically to overwrite the optional arguments passed to the `STAR_ALIGN` module. In this case, the options to be provided to `STAR_ALIGN` have been assigned sensible defaults by the developer(s) in the pipeline's [`modules.config`](https://github.com/nf-core/rnaseq/blob/4c27ef5610c87db00c3c5a3eed10b1d161abf575/conf/modules.config#L70-L74) and can be accessed and customised in the [workflow context](https://github.com/nf-core/rnaseq/blob/4c27ef5610c87db00c3c5a3eed10b1d161abf575/workflows/rnaseq.nf#L201-L204) too before eventually passing them to the sub-workflow as a Groovy map called `star_align_options`. These options will then be propagated from `workflow -> sub-workflow -> module`.
-
-As mentioned at the beginning of this section it may also be necessary for users to overwrite the options passed to modules to be able to customise specific aspects of the way in which a particular tool is executed by the pipeline. Given that all of the default module options are stored in the pipeline's `modules.config` as a [`params` variable](https://github.com/nf-core/rnaseq/blob/4c27ef5610c87db00c3c5a3eed10b1d161abf575/conf/modules.config#L24-L25) it is also possible to overwrite any of these options via a custom config file.
-
-Say for example we want to append an additional, non-mandatory parameter (i.e. `--outFilterMismatchNmax 16`) to the arguments passed to the `STAR_ALIGN` module. Firstly, we need to copy across the default `args` specified in the [`modules.config`](https://github.com/nf-core/rnaseq/blob/4c27ef5610c87db00c3c5a3eed10b1d161abf575/conf/modules.config#L71) and create a custom config file that is a composite of the default `args` as well as the additional options you would like to provide. This is very important because Nextflow will overwrite the default value of `args` that you provide via the custom config.
-
-As you will see in the example below, we have:
-
-* appended `--outFilterMismatchNmax 16` to the default `args` used by the module.
-* changed the default `publish_dir` value to where the files will eventually be published in the main results directory.
-* appended `'bam':''` to the default value of `publish_files` so that the BAM files generated by the process will also be saved in the top-level results directory for the module. Note: `'out':'log'` means any file/directory ending in `out` will now be saved in a separate directory called `my_star_directory/log/`.
-
-```nextflow
-params {
-    modules {
-        'star_align' {
-            args          = "--quantMode TranscriptomeSAM --twopassMode Basic --outSAMtype BAM Unsorted --readFilesCommand zcat --runRNGseed 0 --outFilterMultimapNmax 20 --alignSJDBoverhangMin 1 --outSAMattributes NH HI AS NM MD --quantTranscriptomeBan Singleend --outFilterMismatchNmax 16"
-            publish_dir   = "my_star_directory"
-            publish_files = ['out':'log', 'tab':'log', 'bam':'']
-        }
-    }
-}
-```
-


JoseEspinosa · 2021-11-25T18:43:50Z

nf_core/pipeline-template/nextflow_schema.json

@@ -178,22 +178,6 @@
                    "fa_icon": "fas fa-question-circle",
                    "hidden": true
                },
-                "publish_dir_mode": {


github-actions · 2021-11-25T20:11:18Z

Python linting (`black`) is failing

To keep the code consistent with lots of contributors, we run automated code consistency checks.
To fix this CI test, please run:

Install black: pip install black
Fix formatting errors in your pipeline: black .

Once you push these changes the test should pass, and you can hide this comment 👍

We highly recommend setting up Black in your code editor so that this formatting is done automatically on save. Ask about it on Slack for help!

Thanks again for your contribution!

drpatelh added 15 commits November 25, 2021 15:25

Update module template

9e88474

Update module main.nf

e7b21ea

Bump NF version from 21.04.0 to 21.10.3

f3c6041

Update pipeline template CI to use specified version and latest edge …

6708ec5

…release

Update CHANGELOG

552f2ba

Change syntax in modules.config

26362a8

Remove docs relative to old DSL2 syntax

aa3dee8

Update nextflow.config for new syntax

6b47328

Remve --singularity_pull_docker_container

1490771

Remove addParams call!

175ab50

Remove addParams call!

19f4953

Update container syntax

e4874b5

Remove --publish_dir_mode from pipeline template

dffa555

Artificially update nf-core/modules to fix linting everywhere else

1c7727c

Update modules.json

bc1e63e

drpatelh requested review from JoseEspinosa and grst November 25, 2021 17:08

JoseEspinosa approved these changes Nov 25, 2021

View reviewed changes

drpatelh merged commit 44bc796 into nf-core:dev Nov 25, 2021

This was referenced Nov 26, 2021

Update pipeline template with new, native DSL2 syntax #1290

Closed

Update module template with new, native DSL2 syntax #1289

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update module and pipeline template #1336

Update module and pipeline template #1336

drpatelh commented Nov 25, 2021

JoseEspinosa left a comment •

edited

Loading

JoseEspinosa Nov 25, 2021

JoseEspinosa Nov 25, 2021

github-actions bot commented Nov 25, 2021

Update module and pipeline template #1336

Update module and pipeline template #1336

Conversation

drpatelh commented Nov 25, 2021

JoseEspinosa left a comment • edited Loading

Choose a reason for hiding this comment

JoseEspinosa Nov 25, 2021

Choose a reason for hiding this comment

JoseEspinosa Nov 25, 2021

Choose a reason for hiding this comment

github-actions bot commented Nov 25, 2021

Python linting (black) is failing

JoseEspinosa left a comment •

edited

Loading

Python linting (`black`) is failing