nested nextflow schema params

[mattheww95]

Created a nested nextflow_schema.json

[github-actions]

nf-core lint overall result: Passed ✅ ⚠️

Posted for pipeline commit d944eb8

+| ✅ 146 tests passed       |+
#| ❔  31 tests were ignored |#
!| ❗   2 tests had warnings |!

❗ Test warnings:

• readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
• schema_lint - Schema $id should be https://raw.githubusercontent.com/phac-nml/mikrokondo/master/nextflow_schema.json
  Found https://raw.githubusercontent.com/phac-nml/mikrokondo/main/nextflow_schema.json

❔ Tests ignored:

• files_exist - File is ignored: CODE_OF_CONDUCT.md
• files_exist - File is ignored: assets/nf-core-mikrokondo_logo_light.png
• files_exist - File is ignored: docs/images/nf-core-mikrokondo_logo_light.png
• files_exist - File is ignored: docs/images/nf-core-mikrokondo_logo_dark.png
• files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
• files_exist - File is ignored: .github/workflows/awstest.yml
• files_exist - File is ignored: .github/workflows/awsfulltest.yml
• files_exist - File is ignored: docs/output.md
• files_exist - File is ignored: docs/README.md
• files_exist - File is ignored: docs/usage.md
• nextflow_config - Config variable ignored: manifest.name
• nextflow_config - Config variable ignored: manifest.homePage
• files_unchanged - File does not exist: CODE_OF_CONDUCT.md
• files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
• files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
• files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
• files_unchanged - File does not exist: .github/ISSUE_TEMPLATE/config.yml
• files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/feature_request.yml
• files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
• files_unchanged - File ignored due to lint config: .github/workflows/branch.yml
• files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
• files_unchanged - File ignored due to lint config: assets/email_template.html
• files_unchanged - File ignored due to lint config: assets/email_template.txt
• files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
• files_unchanged - File does not exist: assets/nf-core-mikrokondo_logo_light.png
• files_unchanged - File does not exist: docs/images/nf-core-mikrokondo_logo_light.png
• files_unchanged - File does not exist: docs/images/nf-core-mikrokondo_logo_dark.png
• files_unchanged - File does not exist: docs/README.md
• files_unchanged - File ignored due to lint config: .gitignore or .prettierignore or pyproject.toml
• actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/mikrokondo/mikrokondo/.github/workflows/awstest.yml
• multiqc_config - multiqc_config

✅ Tests passed:

• files_exist - File found: .gitattributes
• files_exist - File found: .gitignore
• files_exist - File found: .nf-core.yml
• files_exist - File found: .editorconfig
• files_exist - File found: .prettierignore
• files_exist - File found: .prettierrc.yml
• files_exist - File found: CHANGELOG.md
• files_exist - File found: CITATIONS.md
• files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
• files_exist - File found: nextflow_schema.json
• files_exist - File found: nextflow.config
• files_exist - File found: README.md
• files_exist - File found: .github/.dockstore.yml
• files_exist - File found: .github/CONTRIBUTING.md
• files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
• files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
• files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
• files_exist - File found: .github/workflows/branch.yml
• files_exist - File found: .github/workflows/ci.yml
• files_exist - File found: .github/workflows/linting_comment.yml
• files_exist - File found: .github/workflows/linting.yml
• files_exist - File found: assets/email_template.html
• files_exist - File found: assets/email_template.txt
• files_exist - File found: assets/sendmail_template.txt
• files_exist - File found: conf/modules.config
• files_exist - File found: conf/test.config
• files_exist - File found: conf/test_full.config
• files_exist - File found: main.nf
• files_exist - File found: assets/multiqc_config.yml
• files_exist - File found: conf/base.config
• files_exist - File found: conf/igenomes.config
• files_exist - File found: modules.json
• files_exist - File found: pyproject.toml
• files_exist - File not found check: Singularity
• files_exist - File not found check: parameters.settings.json
• files_exist - File not found check: pipeline_template.yml
• files_exist - File not found check: .nf-core.yaml
• files_exist - File not found check: bin/markdown_to_html.r
• files_exist - File not found check: conf/aws.config
• files_exist - File not found check: .github/workflows/push_dockerhub.yml
• files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
• files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
• files_exist - File not found check: docs/images/nf-core-mikrokondo_logo.png
• files_exist - File not found check: .markdownlint.yml
• files_exist - File not found check: .yamllint.yml
• files_exist - File not found check: lib/Checks.groovy
• files_exist - File not found check: lib/Completion.groovy
• files_exist - File not found check: lib/Workflow.groovy
• files_exist - File not found check: lib/Utils.groovy
• files_exist - File not found check: lib/WorkflowMain.groovy
• files_exist - File not found check: lib/NfcoreTemplate.groovy
• files_exist - File not found check: lib/WorkflowMikrokondo.groovy
• files_exist - File not found check: lib/nfcore_external_java_deps.jar
• files_exist - File not found check: .travis.yml
• nextflow_config - Config variable found: manifest.nextflowVersion
• nextflow_config - Config variable found: manifest.description
• nextflow_config - Config variable found: manifest.version
• nextflow_config - Config variable found: timeline.enabled
• nextflow_config - Config variable found: trace.enabled
• nextflow_config - Config variable found: report.enabled
• nextflow_config - Config variable found: dag.enabled
• nextflow_config - Config variable found: process.cpus
• nextflow_config - Config variable found: process.memory
• nextflow_config - Config variable found: process.time
• nextflow_config - Config variable found: params.outdir
• nextflow_config - Config variable found: params.input
• nextflow_config - Config variable found: params.validationShowHiddenParams
• nextflow_config - Config variable found: params.validationSchemaIgnoreParams
• nextflow_config - Config variable found: manifest.mainScript
• nextflow_config - Config variable found: timeline.file
• nextflow_config - Config variable found: trace.file
• nextflow_config - Config variable found: report.file
• nextflow_config - Config variable found: dag.file
• nextflow_config - Config variable (correctly) not found: params.nf_required_version
• nextflow_config - Config variable (correctly) not found: params.container
• nextflow_config - Config variable (correctly) not found: params.singleEnd
• nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
• nextflow_config - Config variable (correctly) not found: params.name
• nextflow_config - Config variable (correctly) not found: params.enable_conda
• nextflow_config - Config timeline.enabled had correct value: true
• nextflow_config - Config report.enabled had correct value: true
• nextflow_config - Config trace.enabled had correct value: true
• nextflow_config - Config dag.enabled had correct value: true
• nextflow_config - Config dag.file ended with .html
• nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
• nextflow_config - Config manifest.version ends in dev: 1.0dev
• nextflow_config - Config params.custom_config_version is set to master
• nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
• nextflow_config - Lines for loading custom profiles found
• nextflow_config - nextflow.config contains configuration profile test
• nextflow_config - Config default value correct: params.platform= illumina
• nextflow_config - Config default value correct: params.long_read_opt= nanopore
• nextflow_config - Config default value correct: params.dehosting_idx= ./databases/PhiPacHum_m2.idx
• nextflow_config - Config default value correct: params.mash_sketch= ./databases/GTDBSketch_20231003.msh
• nextflow_config - Config default value correct: params.bakta_db= ./databases/db-light
• nextflow_config - Config default value correct: params.kraken2_db= ./databases/k2_standard_20220607/
• nextflow_config - Config default value correct: params.custom_config_version= master
• nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
• nextflow_config - Config default value correct: params.max_cpus= 16
• nextflow_config - Config default value correct: params.max_memory= 2000.GB
• nextflow_config - Config default value correct: params.max_time= 240.h
• nextflow_config - Config default value correct: params.publish_dir_mode= copy
• nextflow_config - Config default value correct: params.validate_params= true
• nextflow_config - Config default value correct: params.tracedir= null/pipeline_info
• nextflow_config - Config default value correct: params.stage_in_mode= symlink
• nextflow_config - Config default value correct: params.validationS3PathCheck= true
• nextflow_config - Config default value correct: params.skip_ont_header_cleaning= true
• nextflow_config - Config default value correct: params.fp_average_quality= 25
• nextflow_config - Config default value correct: params.fp_cut_tail_mean_quality= 15
• nextflow_config - Config default value correct: params.fp_cut_tail_window_size= 4
• nextflow_config - Config default value correct: params.fp_complexity_threshold= 20
• nextflow_config - Config default value correct: params.fp_qualified_phred= 15
• nextflow_config - Config default value correct: params.fp_unqualified_precent_limit= 40
• nextflow_config - Config default value correct: params.fp_polyg_min_len= 10
• nextflow_config - Config default value correct: params.fp_polyx_min_len= 10
• nextflow_config - Config default value correct: params.fp_illumina_length_min= 35
• nextflow_config - Config default value correct: params.fp_illumina_length_max= 400
• nextflow_config - Config default value correct: params.fp_single_end_length_min= 1000
• nextflow_config - Config default value correct: params.target_depth= 100
• nextflow_config - Config default value correct: params.min_reads= 1000
• nextflow_config - Config default value correct: params.ba_min_conting_length= 200
• nextflow_config - Config default value correct: params.qt_min_contig_length= 1000
• nextflow_config - Config default value correct: params.mh_min_kmer= 10
• nextflow_config - Config default value correct: params.flye_read_type= hq
• files_unchanged - .gitattributes matches the template
• files_unchanged - .prettierrc.yml matches the template
• files_unchanged - .github/.dockstore.yml matches the template
• files_unchanged - .github/workflows/linting_comment.yml matches the template
• actions_ci - '.github/workflows/ci.yml' is triggered on expected events
• actions_ci - '.github/workflows/ci.yml' checks minimum NF version
• readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
• pipeline_todos - No TODO strings found
• pipeline_name_conventions - Name adheres to nf-core convention
• template_strings - Did not find any Jinja template strings (192 files)
• schema_lint - Schema lint passed
• schema_lint - Input mimetype lint passed: 'text/csv'
• schema_params - Schema matched params returned from nextflow config
• system_exit - No System.exit calls found
• actions_schema_validation - Workflow validation passed: doc.yml
• actions_schema_validation - Workflow validation passed: linting_comment.yml
• actions_schema_validation - Workflow validation passed: branch.yml
• actions_schema_validation - Workflow validation passed: ci.yml
• actions_schema_validation - Workflow validation passed: linting.yml
• merge_markers - No merge markers found in pipeline files
• modules_json - Only installed modules found in modules.json
• modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

• nf-core/tools version 2.13.1
• Run at 2024-03-20 17:03:46

[apetkau]

This looks amazing @mattheww95 . Thanks so much for all your hard work. I do have two comments for you.

[apetkau]

I have fixed up the tests and I have made some additional changes. The main issue is that Nextflow JSON Schemas don't support nested parameters, so we have to remove these from the JSON schema files (sorry about getting mixed up about that and thinking nested parameters would just be ignored).

So, I have made the following changes:

• Removed nextflow_schema.json and re-generated it from the config file using nf-core schema build
  • Re-structured schema to group common parameters together and added descriptions and appropriate minimums for integer parameters.
• Added test cases to verify that validation is happening for parameters and passing incorrect values fails the pipeline with an appropriate error message.
• Added tests to make sure that the identified species is set appropriately

I also tested how this would look in IRIDA Next (see below):

[mattheww95]

Looks great! I am glad we got everything figured out!