Refactor the pipeline and download dbs code. #9

mberacochea · 2024-12-18T16:48:31Z

This ended up being a massive refactoring of the pipeline.

Some of the changes:

Remove the bash script to download the databases - replaced with a subworkflow
Swapped the bespoke fastp module with the one from nf-core
Move things around based on the linter errors (vscode)
Adjusted the parameters - based on db download sciprts
Formatted code code
Added some test reference dbs (this still needs some work)

I'm running some tests...

This ended up being a massive refactoring of the pipeline. Some of the changes: - Remove the bash script to download the databases - replaced with a subworkflow - Swapped the bespoke fastp module with the one from nf-core - Move things around based on the linter errors (vscode) - Adjusted the parameters - based on db download sciprts - Formatted code code - Added some test reference dbs (this still needs some work)

github-actions · 2024-12-18T16:50:00Z

`nf-core pipelines lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 3b4a3cb

+| ✅ 127 tests passed       |+
#| ❔  47 tests were ignored |#
!| ❗  10 tests had warnings |!

❗ Test warnings:

files_exist - File not found: conf/igenomes_ignored.config
files_exist - File not found: .github/workflows/awstest.yml
files_exist - File not found: .github/workflows/awsfulltest.yml
files_exist - File not found: ro-crate-metadata.json
nextflow_config - nf-validation has been detected in the pipeline. Please migrate to nf-schema: https://nextflow-io.github.io/nf-schema/latest/migration_guide/
nextflow_config - Config manifest.version should end in dev: 1.1.0
readme - README did not have a Nextflow minimum version badge.
pipeline_todos - TODO string in WorkflowShallowmapping.groovy: Optionally add in-text citation tools to this list.
pipeline_todos - TODO string in WorkflowMain.groovy: Add Zenodo DOI for pipeline after first release
nfcore_yml - nf-core version not set in .nf-core.yml

❔ Tests ignored:

files_exist - File is ignored: CODE_OF_CONDUCT.md
files_exist - File is ignored: CHANGELOG.md
files_exist - File is ignored: assets/nf-core-shallowmapping_logo_light.png
files_exist - File is ignored: docs/
files_exist - File is ignored: docs/output.md
files_exist - File is ignored: docs/README.md
files_exist - File is ignored: docs/usage.md
files_exist - File is ignored: docs/images/nf-core-shallowmapping_logo_dark.png
files_exist - File is ignored: docs/images/nf-core-shallowmapping_logo_light.png
files_exist - File is ignored: pyproject.toml
files_exist - File is ignored: .gitignore
files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
files_exist - File is ignored: .github/workflows/*
files_exist - File is ignored: .github/workflows/branch.yml
files_exist - File is ignored: .github/CONTRIBUTING.md
files_exist - File is ignored: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File is ignored: conf/igenomes.config
files_exist - File is ignored: conf/igenomes.config
files_exist - File is ignored: conf/test.config
files_exist - File is ignored: conf/test_full.config
files_exist - File is ignored: .github/workflows/ci.yml
files_exist - File is ignored: lib/Utils.groovy
files_exist - File is ignored: lib/WorkflowMain.groovy
files_exist - File is ignored: lib/NfcoreTemplate.groovy
files_exist - File is ignored: lib/WorkflowShallowmapping.groovy
files_exist - File is ignored: lib/nfcore_external_java_deps.jar
nextflow_config - Config variable ignored: manifest.name
nextflow_config - Config variable ignored: manifest.homePage
nextflow_config - Config variable ignored: process.cpus
nextflow_config - Config variable ignored: process.memory
nextflow_config - Config variable ignored: process.time
nextflow_config - Config variable ignored: params.max_cpus
files_unchanged - File ignored due to lint config: CODE_OF_CONDUCT.md
files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
files_unchanged - File does not exist: .github/ISSUE_TEMPLATE/config.yml
files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
files_unchanged - File ignored due to lint config: .github/workflows/branch.yml
files_unchanged - File ignored due to lint config: .github/workflows/linting_comment.yml
files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
files_unchanged - File ignored due to lint config: assets/nf-core-shallowmapping_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-shallowmapping_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-shallowmapping_logo_dark.png
files_unchanged - File does not exist: docs/README.md
files_unchanged - File ignored due to lint config: .gitignore or .prettierignore
actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/shallowmapping/shallowmapping/.github/workflows/awstest.yml

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CITATIONS.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: conf/modules.config
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-shallowmapping_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: .travis.yml
nextflow_config - Found nf-validation plugin
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.outdir= results
nextflow_config - Config default value correct: params.max_cpus= 16
nextflow_config - Config default value correct: params.max_memory= 128.GB
nextflow_config - Config default value correct: params.max_time= 240.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
readme - README Zenodo placeholder was replaced with DOI.
plugin_includes - No wrong validation plugin imports have been found
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (0 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: release-announcments.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
base_config - CUSTOM_DUMPSOFTWAREVERSIONS found in conf/base.config and Nextflow scripts.
modules_config - conf/modules.config found and not ignored.
modules_config - DOWNLOAD_MGNIFY_GENOMES_REFERENCE_DBS found in conf/modules.config and Nextflow scripts.
modules_config - DOWNLOAD_DRAM_DB found in conf/modules.config and Nextflow scripts.
modules_config - DOWNLOAD_HUMAN_PHIX_BWAMEM2_INDEX found in conf/modules.config and Nextflow scripts.
modules_config - FASTP found in conf/modules.config and Nextflow scripts.
modules_config - FASTQC found in conf/modules.config and Nextflow scripts.
modules_config - MULTIQC found in conf/modules.config and Nextflow scripts.
modules_config - SOURMASH_GATHER found in conf/modules.config and Nextflow scripts.
modules_config - ALIGN_BWAMEM2 found in conf/modules.config and Nextflow scripts.
modules_config - POSTPROC_SOURMASHTAXO found in conf/modules.config and Nextflow scripts.
modules_config - POSTPROC_BWATAXO found in conf/modules.config and Nextflow scripts.
modules_config - POSTPROC_FUNCTIONSPRED found in conf/modules.config and Nextflow scripts.
modules_config - DRAM_DISTILL found in conf/modules.config and Nextflow scripts.
modules_config - KEGG_COMPLETENESS found in conf/modules.config and Nextflow scripts.
modules_config - KEGG_SPECIES found in conf/modules.config and Nextflow scripts.
modules_config - POSTPROC_INTEGRATOR found in conf/modules.config and Nextflow scripts.
modules_config - CUSTOM_DUMPSOFTWAREVERSIONS found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline

Run details

nf-core/tools version 3.1.0
Run at 2024-12-19 15:30:02

mberacochea · 2024-12-18T22:52:25Z

Whole pipeline test yey

ochkalova

Nicely done, Martin
I don't have much to say, just two questions 🙂

conf/codon.config

conf/base.config

mberacochea requested review from Ales-ibt, KateSakharova and ochkalova December 18, 2024 16:48

mberacochea self-assigned this Dec 18, 2024

mberacochea added 4 commits December 18, 2024 16:52

Fix lint issue

c25db16

Fix the catalogues URLs

2a338d1

Fix DRAM distill and re-enable the tests.

522389d

Change the test reads urls to the github ones

992df76

mberacochea added 2 commits December 19, 2024 10:51

Fix the download sbfw and adjust the test profile reads

cd24908

Adjust the test samplesheet path

42ccd95

ochkalova approved these changes Dec 19, 2024

View reviewed changes

conf/codon.config Show resolved Hide resolved

conf/base.config Outdated Show resolved Hide resolved

Adjust based on suggestions

3b4a3cb

mberacochea merged commit 72ccfb8 into dev Dec 20, 2024
4 checks passed

mberacochea deleted the update/dbs_download branch December 20, 2024 10:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor the pipeline and download dbs code. #9

Refactor the pipeline and download dbs code. #9

mberacochea commented Dec 18, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

mberacochea commented Dec 18, 2024 •

edited

Loading

ochkalova left a comment

Refactor the pipeline and download dbs code. #9

Refactor the pipeline and download dbs code. #9

Conversation

mberacochea commented Dec 18, 2024 • edited Loading

github-actions bot commented Dec 18, 2024 • edited Loading

nf-core pipelines lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

mberacochea commented Dec 18, 2024 • edited Loading

ochkalova left a comment

Choose a reason for hiding this comment

mberacochea commented Dec 18, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024 •

edited

Loading

`nf-core pipelines lint` overall result: Passed ✅ ⚠️

mberacochea commented Dec 18, 2024 •

edited

Loading