Migration to topic channels

[edmundmiller]

Starting an issue to track. Probably gonna need a hackathon for this 🙃
docs

Tasks

Beta Give feedback

1. Update tools linting
2. PoC PR
3. Bump Nextflow version in pipelines, template, and modules CI
4. Update pipeline code to use topics
5. Create a "validate versions.yml" helper function

[edmundmiller]

https://nfcore.slack.com/archives/C043FMKUNLB/p1701792739273019

[mahesh-panchal]

Just to reiterate what I said there, it seems topic makes a global queue type channel that can be fed into. It's useful for emitting versions but not for something like emitting say bam and bai files together because of process aliasing, etc.

[edmundmiller]

Yeah these will just be used for the versions files (though maybe we can start adding a multiqc topic channel 👀)

[stevekm]

I had a demo of a proposed methodology for using the topic channels for this here;

https://github.com/stevekm/nextflow-demos/tree/master/topic-channels-versions

additionally, I am working on a Nextflow plugin that will be able to directly convert the topic channel versions outputs to the MultiQC Software Versions table. So we could skip the individual versions-YAML files and go directly to the final MultiQC table.

Also I had a Discussion here about standardizations for the schema of the topic channels to be used for this

nextflow-io/nextflow#4925

if we had a standardized schema to use for "versions" then it would be easier for everyone to align on consistent handling methods. In particular, I wanted to start including the "container" in the versions output, I am not sure that its currently being captured in the legacy versions YAML format

also there was a related issue here nextflow-io/nextflow#4934

[stevekm]

@maxulysse

[ewels]

Plan after discussion with @mashehu and @mirpedrol:

• Progressively move to topics, break dependency with Use eval to collect version #5834
• Use a topic name versions_file with old versions.yml path output style
• Use a topic name versions for a new eval syntax
• Have pipeline-level logic to load the old files and process into a string that matches the new eval syntax
• Pipeline logic to take the new style strings and generate the required YAML files for reporting

This way we can progressively switch to the new eval syntax in modules, which will be a rather large and manual effort.

Same with topics - the old channels will remain for now with the same channel names, so the old syntax should continue to work.

[ewels]

Should also consider other usage of topic channels whilst we're here.

One that springs to mind is outputs for MultiQC. These are often emitted from many processes and collected at the end in a similar way to versions. Could have a multiqc topic for any outputs that should be used for that purpose.