Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move binning entry test to binrefinement test, and fix binning entry+ concoct bug #473

Merged
merged 6 commits into from
Jul 13, 2023
Merged

Move binning entry test to binrefinement test, and fix binning entry+ concoct bug #473

merged 6 commits into from
Jul 13, 2023

Conversation

prototaxites
Copy link
Contributor

@prototaxites prototaxites commented Jul 10, 2023 •
edited
Loading

test_binrefinement takes a long time to run - why not skip a whole bunch of steps!

Edit: In doing this, found a binning entry bug - the pipeline works internally with uncompressed assemblies, but I supplied the assemblies on the test-data site as gzipped (which is likely what users will have from a previous mag run). Added a step in setting up the assemblies channel which checks binning_entry fasta files for gzippedness (by file extension), and uncompresses them if so. Updated the docs accordingly.

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs- [ ] If necessary, also make a PR on the nf-core/mag branch on the nf-core/test-datasets repository.
  • Make sure your code lints (nf-core lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

@github-actions
Copy link

github-actions bot commented Jul 10, 2023 •
edited
Loading

nf-core lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 7520020

+| ✅ 156 tests passed       |+
#| ❔   1 tests were ignored |#
!| ❗   1 tests had warnings |!

❗ Test warnings:

  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline

❔ Tests ignored:

  • files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy

✅ Tests passed:

Run details

  • nf-core/tools version 2.9
  • Run at 2023-07-11 13:28:55

@prototaxites
Copy link
Contributor Author

prototaxites commented Jul 10, 2023 •
edited
Loading

Error running the CI tests:

ERROR ~ Error executing process > 'NFCORE_MAG:MAG:KRAKEN2 (test_minigut-minigut_kraken)'

Caused by:
  java.util.NoSuchElementException -- Check script './workflows/../modules/local/kraken2.nf' at line: 20

Source block:
  def input = meta.single_end ? "\"${reads}\"" :  "--paired \"${reads[0]}\" \"${reads[1]}\""
  prefix = task.ext.prefix ?: "${meta.id}"
  """
      kraken2 \
          --report-zero-counts \
          --threads ${task.cpus} \
          --db database \
          --report ${prefix}.kraken2_report.txt \
          $input \
          > kraken2.kraken
      cat kraken2.kraken | cut -f 2,3 > results.krona
  
      cat <<-END_VERSIONS > versions.yml
      "${task.process}":
          kraken2: \$(echo \$(kraken2 --version 2>&1) | sed 's/^.*Kraken version //' | sed 's/ Copyright.*//')
      END_VERSIONS
      """

Same error as #470 (comment), this time with Kraken2 instead of Centrifuge. Again, line 20 in kraken2.nf is prefix = task.ext.prefix ?: "${meta.id}" - something is unsafely modifying the meta map is my best guess!

@prototaxites
Copy link
Contributor Author

Another error:

ERROR ~ Error executing process > 'NFCORE_MAG:MAG:BINNING:FASTA_BINNING_CONCOCT:CONCOCT_CUTUPFASTA (test_minigut)'

Caused by:
  Process `NFCORE_MAG:MAG:BINNING:FASTA_BINNING_CONCOCT:CONCOCT_CUTUPFASTA (test_minigut)` terminated with an error exit status (1)

Command executed:

  cut_up_fasta.py \
      SPAdes-test_minigut_contigs.fasta.gz \
       \
      -b SPAdes-CONCOCT-test_minigut.bed \
      > SPAdes-CONCOCT-test_minigut.fasta
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_MAG:MAG:BINNING:FASTA_BINNING_CONCOCT:CONCOCT_CUTUPFASTA":
      concoct: $(echo $(concoct --version 2>&1) | sed 's/concoct //g' )
  END_VERSIONS

Command exit status:
  1

Command output:
  (empty)

Command error:
  Unable to find image 'quay.io/biocontainers/concoct:1.1.0--py38h7be5676_2' locally
  1.1.0--py38h7be5676_2: Pulling from biocontainers/concoct
  a3ed95caeb02: Already exists
  77c6c00e8b61: Already exists
  3aaade50789a: Already exists
  00cf8b9f3d2a: Already exists
  7ff999a2256f: Already exists
  d2ba336f2e44: Already exists
  dfda3e01f2b6: Already exists
  a3ed95caeb02: Already exists
  10c3bb32200b: Already exists
  6d0ba9e0d1d0: Pulling fs layer
  6d0ba9e0d1d0: Verifying Checksum
  6d0ba9e0d1d0: Download complete
  6d0ba9e0d1d0: Pull complete
  Digest: sha256:d608a4ac91935bd49aaee2f4d6b3277faf0dfcc1c74b48538b0b8a7c18647b8b
  Status: Downloaded newer image for quay.io/biocontainers/concoct:1.1.0--py38h7be5676_2
  Traceback (most recent call last):
    File "/usr/local/bin/cut_up_fasta.py", line 59, in <module>
      cut_up_fasta(args.contigs, args.chunk_size, args.overlap_size, args.merge_last, args.bedfile)
    File "/usr/local/bin/cut_up_fasta.py", line 16, in cut_up_fasta
      for record in SeqIO.parse(ff, "fasta"):
    File "/usr/local/lib/python3.8/site-packages/Bio/SeqIO/Interfaces.py", line 68, in __next__
      return next(self.records)
    File "/usr/local/lib/python3.8/site-packages/Bio/SeqIO/FastaIO.py", line 199, in iterate
      for title, sequence in SimpleFastaParser(handle):
    File "/usr/local/lib/python3.8/site-packages/Bio/SeqIO/FastaIO.py", line 47, in SimpleFastaParser
      for line in handle:
    File "/usr/local/lib/python3.8/codecs.py", line 322, in decode
      (result, consumed) = self._buffer_decode(data, self.errors, final)
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

Dumping here in case I lose it when the CI re-runs. Looking at the module, CONCOCT needs uncompressed assemblies - the binning entry spreadsheet supplies them as gzipped.

@prototaxites prototaxites changed the title Move binning entry test to binrefinement test Move binning entry test to binrefinement test, and fix binning entry+ concoct bug Jul 11, 2023
@prototaxites prototaxites mentioned this pull request Jul 11, 2023
Closed
jfy133
jfy133 approved these changes Jul 13, 2023
View reviewed changes
Copy link
Member

@jfy133 jfy133 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice idea! One question but not a blocker if it's working!

workflows/mag.nf Show resolved Hide resolved
@prototaxites prototaxites merged commit 87761e0 into nf-core:dev Jul 13, 2023
@prototaxites prototaxites deleted the assembly_entry_test branch July 17, 2023 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jfy133 jfy133 jfy133 approved these changes

@d4straub d4straub Awaiting requested review from d4straub

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@prototaxites @jfy133