Collect Versions for individual processes

[apeltzer]

New feature

Nextflow should be able to collect versions of executed commands in a sophisticated manner. We have a small python script in use at nf-core that works reasonably well for "one-container-per-pipeline" workflows, but that approach is a bit more tricky in the upcoming modularized world, where in theory each process should have such a command.

Usage scenario

Each process will have an extra section that collects version information. This is optional, users don't have to do it but will be executed by Nextflow whenever the process is executed and a label, the command and a regular expression that extracts the version string is specified. (see below for basic example).

Suggest implementation

Example:

Given foo --version produces a string as output: version: 2.3, the below mentioned snippet should collect the version and save it in an intermediate Map<k,v>() pair and later output this in JSON/YAML format.

process foo {
[.....]
version: 'foo', 'foo --version', '\d+'

[.....]

script:
"""
foo $fasta $bam 
"""
}

The outputted JSON/YAML would then have a basic format that has the name, version tuples present.

Benefits:

• Clear versions handling for future applications (e.g. monitoring which tool was used for each step in a pipeline for example)
• Works very well for "one container per process" when DSLv2 modules is coming
• We can drop some boilerplate code from nf-core template ;-)
• Benefits are for all users of Nextflow, not just nf-core people

I'd really love to get this going and implement it myself but would appreciate some small hints on where I'd need to add this to the required scopes ideally. E.g. where to plug this in - then I can give this a go (already started drafting, but waiting for feedback from @pditommaso with this now) 👍

@ewels @maxulysse @drpatelh might be interested in seeing this :-)

[ewels]

See #879 😉

[apeltzer]

Duplicate then :-D