Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

failure on AMRfinder (core dump) #144

Closed
SeanSierra-Patev opened this issue Feb 27, 2024 · 21 comments · Fixed by #146
Closed

failure on AMRfinder (core dump) #144

SeanSierra-Patev opened this issue Feb 27, 2024 · 21 comments · Fixed by #146
Assignees
@jvhagey
Labels
Bug: System Specific Something isn't working, specific to certain systems

Comments

@SeanSierra-Patev
Copy link

Failure on AMRfinder run, error is printed below

Feb-27 14:56:01.558 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=PHOENIX:PHOENIX_EXTERNAL:AMRFINDERPLUS_RUN (RISHL24AMR036); work-dir=/home/user1/Workdir/2_2024_mixed/run2/work/b4/83799f37b3a73cf7129874c08fd7bd
  error [nextflow.exception.ProcessFailedException]: Process `PHOENIX:PHOENIX_EXTERNAL:AMRFINDERPLUS_RUN (RISHL24AMR036)` terminated with an error exit status (1)
Feb-27 14:56:01.571 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'PHOENIX:PHOENIX_EXTERNAL:AMRFINDERPLUS_RUN (RISHL24AMR036)'

Caused by:
  Process `PHOENIX:PHOENIX_EXTERNAL:AMRFINDERPLUS_RUN (RISHL24AMR036)` terminated with an error exit status (1)

Command executed:

  if [[ RISHL24AMR036.filtered.scaffolds.fa.gz = *.gz ]]; then
      NUC_FNAME=$(basename RISHL24AMR036.filtered.scaffolds.fa.gz .gz)
      gzip -c -d RISHL24AMR036.filtered.scaffolds.fa.gz > $NUC_FNAME
  else
      NUC_FNAME = RISHL24AMR036.filtered.scaffolds.fa.gz
  fi
  
  # decompress the amrfinder database
  tar xzvf amrfinderdb_v3.11_20230417.1.tar.gz
  
  amrfinder \
      --nucleotide $NUC_FNAME \
      --protein RISHL24AMR036.faa \
      --gff RISHL24AMR036.gff \
      --annotation_format prokka \
      --mutation_all RISHL24AMR036_all_mutations.tsv \
      --organism Klebsiella_oxytoca \
      --plus \
      --database amrfinderdb_v3.11_20230417.1 \
      --threads 6 > RISHL24AMR036_all_genes.tsv
  
  sed -i '1s/ /_/g' RISHL24AMR036_all_genes.tsv
  
  cat <<-END_VERSIONS > versions.yml
  "PHOENIX:PHOENIX_EXTERNAL:AMRFINDERPLUS_RUN":
      amrfinderplus: $(amrfinder --version)
      amrfinderplus_db_version: $(head amrfinderdb_v3.11_20230417.1/version.txt)
  END_VERSIONS

Command exit status:
  1

Command output:
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Neisseria_gonorrhoeae.nhr
  amrfinderdb_v3.11_20230417.1/AMRProt-susceptible.tab
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Neisseria_gonorrhoeae.not
  amrfinderdb_v3.11_20230417.1/AMR_CDS.ndb
  amrfinderdb_v3.11_20230417.1/AMRProt.ptf
  amrfinderdb_v3.11_20230417.1/AMRProt.psq
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Escherichia.nto
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Enterococcus_faecium.tab
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Streptococcus_pneumoniae.nto
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Salmonella.njs
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Klebsiella_oxytoca.tab
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Salmonella.nto
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Campylobacter.not
  amrfinderdb_v3.11_20230417.1/AMRProt
  amrfinderdb_v3.11_20230417.1/AMRProt.phr
  amrfinderdb_v3.11_20230417.1/AMR.LIB.h3p
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Escherichia.njs
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Enterococcus_faecalis.nhr
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Campylobacter.nto
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Clostridioides_difficile.nsq
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Enterococcus_faecium.nhr
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Clostridioides_difficile.not
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Escherichia.ntf
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Neisseria_gonorrhoeae.njs
  amrfinderdb_v3.11_20230417.1/AMR_CDS.nsq
  amrfinderdb_v3.11_20230417.1/AMRProt.pto
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Streptococcus_pneumoniae.ndb
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Enterococcus_faecium.nin
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Enterococcus_faecium.nsq
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Enterococcus_faecium.ntf
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Staphylococcus_aureus.tab
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Enterococcus_faecium.nto
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Neisseria_gonorrhoeae.tab
  amrfinderdb_v3.11_20230417.1/AMRProt.pjs
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Campylobacter.ndb
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Campylobacter.tab
  amrfinderdb_v3.11_20230417.1/AMR_CDS.nto
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Staphylococcus_aureus.not
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Enterococcus_faecalis.ndb
  amrfinderdb_v3.11_20230417.1/AMRProt.pin
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Klebsiella_oxytoca.nhr
  amrfinderdb_v3.11_20230417.1/AMR_CDS.nhr
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Streptococcus_pneumoniae.njs
  amrfinderdb_v3.11_20230417.1/AMR.LIB.h3m
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Clostridioides_difficile.tab
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Clostridioides_difficile.nhr
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Streptococcus_pneumoniae.nin
  amrfinderdb_v3.11_20230417.1/AMR.LIB
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Clostridioides_difficile.ndb
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Neisseria_gonorrhoeae.nsq

Command error:
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Escherichia.ntf
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Neisseria_gonorrhoeae.njs
  amrfinderdb_v3.11_20230417.1/AMR_CDS.nsq
  amrfinderdb_v3.11_20230417.1/AMRProt.pto
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Streptococcus_pneumoniae.ndb
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Enterococcus_faecium.nin
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Enterococcus_faecium.nsq
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Enterococcus_faecium.ntf
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Staphylococcus_aureus.tab
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Enterococcus_faecium.nto
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Neisseria_gonorrhoeae.tab
  amrfinderdb_v3.11_20230417.1/AMRProt.pjs
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Campylobacter.ndb
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Campylobacter.tab
  amrfinderdb_v3.11_20230417.1/AMR_CDS.nto
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Staphylococcus_aureus.not
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Enterococcus_faecalis.ndb
  amrfinderdb_v3.11_20230417.1/AMRProt.pin
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Klebsiella_oxytoca.nhr
  amrfinderdb_v3.11_20230417.1/AMR_CDS.nhr
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Streptococcus_pneumoniae.njs
  amrfinderdb_v3.11_20230417.1/AMR.LIB.h3m
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Clostridioides_difficile.tab
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Clostridioides_difficile.nhr
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Streptococcus_pneumoniae.nin
  amrfinderdb_v3.11_20230417.1/AMR.LIB
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Clostridioides_difficile.ndb
  amrfinderdb_v3.11_20230417.1/AMR_DNA-Neisseria_gonorrhoeae.nsq
  Running: amrfinder --nucleotide RISHL24AMR036.filtered.scaffolds.fa --protein RISHL24AMR036.faa --gff RISHL24AMR036.gff --annotation_format prokka --mutation_all RISHL24AMR036_all_mutations.tsv --organism Klebsiella_oxytoca --plus --database amrfinderdb_v3.11_20230417.1 --threads 6
  Software directory: '/amrfinder/'
  Software version: 3.11.11
  Database directory: 'amrfinderdb_v3.11_20230417.1'
  Database version: 2023-04-17.1
  AMRFinder combined translated and protein and mutation search
  Running blastp
  Running hmmsearch
  Running tblastn
  Running blastn
  
  *** ERROR ***
  '/ncbi-blast-2.13.0+/bin/blastn'  -query 'RISHL24AMR036.filtered.scaffolds.fa' -db /tmp/amrfinder.Rky7fl/db/AMR_DNA-Klebsiella_oxytoca -evalue 1e-20  -dust no  -max_target_seqs 10000    -num_threads 2  -mt_mode 1 -outfmt '6 qseqid sseqid qstart qend qlen sstart send slen qseq sseq' -out /tmp/amrfinder.Rky7fl/blastn > /tmp/amrfinder.Rky7fl/log 2> /tmp/amrfinder.Rky7fl/blastn-err
  status = 35584
  Segmentation fault (core dumped)
  
  HOSTNAME: a908a61be5c2
  SHELL: ?
  PWD: /home/user1/Workdir/2_2024_mixed/run2/work/b4/83799f37b3a73cf7129874c08fd7bd
  PATH: /amrfinder:/ncbi-blast-2.13.0+/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/user1/.nextflow/assets/cdcgov/phoenix/bin
  Progam name:  amrfinder
  Command line: amrfinder --nucleotide RISHL24AMR036.filtered.scaffolds.fa --protein RISHL24AMR036.faa --gff RISHL24AMR036.gff --annotation_format prokka --mutation_all RISHL24AMR036_all_mutations.tsv --organism Klebsiella_oxytoca --plus --database amrfinderdb_v3.11_20230417.1 --threads 6

Work dir:
  /home/user1/Workdir/2_2024_mixed/run2/work/b4/83799f37b3a73cf7129874c08fd7bd

I've repeated it twice (with different version of PhoeNIX) and gotten the same result both times.
@jvhagey
Copy link
Contributor

jvhagey commented Feb 27, 2024

I think this is a resource problem like this? #143 can you up the resources and let me know what level works?

@SeanSierra-Patev
Copy link
Author

overall, with --max_memory / --max_cpus or in the conf for this tool specifically? Because I thought the default was 128G which is all I've got

@jvhagey
Copy link
Contributor

jvhagey commented Feb 27, 2024

No, the steps are individually set to not over use resources. For AMRFinder the defaults is 4GB can you up in this section of the code and see if it resolves? It is set to try again and double the memory if it runs out, but maybe something about how AMRFinder is failing its not seeing it as a memory problem? If its a sample specific issue I can also try running it and debugging if you send it to [email protected]

@SeanSierra-Patev
Copy link
Author

Ok sure, I'll give that a go and get back to you. I have something else running at the moment but I will give it a try first thing in the AM

@SeanSierra-Patev
Copy link
Author

increasing to 8 and then 16 did not resolve the core dump unfortunately

@jvhagey
Copy link
Contributor

jvhagey commented Mar 12, 2024

Updating the issue:
@SeanSierra-Patev sent me the files and they ran to completion just fine and it doesn't seem that lack of memory would be the issue here.
image

The only reference I can find is this recent issue ncbi/amr#118 which points to the sequence line length being a possible issue.

Can you try running phx v2.1.1-dev, it has an updated amrfinder container, but the blastn isn't a different version? I don't actually know if this is phx specific or an amrfinder/blastn issue. So can you also try running amrfinder outside of the pipeline?

@jvhagey jvhagey self-assigned this Mar 12, 2024
@tang27abu
Copy link

Hi I encountered the same issue,

  *** ERROR ***
  '/ncbi-blast-2.13.0+/bin/blastn'  -query 'AR-0074.filtered.scaffolds.fa' -db /tmp/amrfinder.hRw81Y/db/AMR_DNA-Klebsiella_oxytoca -evalue 1e-20  -dust no  -max_target_seqs 10000    -num_threads 2  -mt_mode 1 -outfmt '6 qseqid sseqid qstart qend qlen sstart send slen qseq sseq' -out /tmp/amrfinder.hRw81Y/blastn > /tmp/amrfinder.hRw81Y/log 2> /tmp/amrfinder.hRw81Y/blastn-err
  status = 35584
  Segmentation fault (core dumped)
  
  HOSTNAME: 98e20ea005df
  SHELL: ?
  PWD: /scratch/zl7w2/tools/phoenix/work/fb/1255d78d434d0186df9a7d889b8517
  PATH: /amrfinder:/ncbi-blast-2.13.0+/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/scratch/zl7w2/tools/phoenix/bin
  Progam name:  amrfinder
  Command line: amrfinder --nucleotide AR-0074.filtered.scaffolds.fa --protein AR-0074.faa --gff AR-0074.gff --annotation_format prokka --mutation_all AR-0074_all_mutations.tsv --organism Klebsiella_oxytoca --plus --database amrfinderdb_v3.11_20230417.1 --threads 6

Work dir:
  /scratch/zl7w2/tools/phoenix/work/fb/1255d78d434d0186df9a7d889b8517

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details

How can I test the data with the phoenix dev version?

@jvhagey
Copy link
Contributor

jvhagey commented Mar 12, 2024 •
edited
Loading

@tang27abu, can you provide more detail on this sample? Is it an AR Bank sample?

Here is the command:
nextflow run cdcgov/phoenix -r v2.1.1-dev -latest -profile singularity -entry PHOENIX --input samplesheet.csv --kraken2db $kraken2db

@tang27abu
Copy link

@tang27abu, can you provide more detail on this sample? Is it an AR Bank sample?

Here is the command: nextflow run cdcgov/phoenix -r v2.1.1-dev -latest -profile singularity -entry PHOENIX --input samplesheet.csv --kraken2db $kraken2db

I was trying to run with this command nextflow run cdcgov/phoenix -r v2.1.1-dev /scratch/zl7w2/tools/phoenix/main.nf -profile docker -entry PHOENIX --input /scratch/zl7w2/Matt/MUCRO4/reads/metadata.csv --kraken2db /scratch/zl7w2/tools/phoenix/k2db
And get the error below:

[50/44fca7] process > PHOENIX:PHOENIX_EXTERNAL:GET_TAXA_FOR_AMRFINDER (AR-0074)                             [100%] 4 of 4
[66/48cce1] process > PHOENIX:PHOENIX_EXTERNAL:AMRFINDERPLUS_RUN (AR-0074)                                  [100%] 4 of 4, failed: 1
[9d/a4abd0] process > PHOENIX:PHOENIX_EXTERNAL:CALCULATE_ASSEMBLY_RATIO (AR-0074)                           [100%] 4 of 4
[e6/2af222] process > PHOENIX:PHOENIX_EXTERNAL:GENERATE_PIPELINE_STATS_WF:GENERATE_PIPELINE_STATS (AR-0034) [100%] 3 of 3
[f2/01cce4] process > PHOENIX:PHOENIX_EXTERNAL:CREATE_SUMMARY_LINE (AR-0034)                                [100%] 3 of 3
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:FETCH_FAILED_SUMMARIES                                       -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:GATHER_SUMMARY_LINES                                         -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:GRIPHIN                                                      -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:CUSTOM_DUMPSOFTWAREVERSIONS                                  -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:MULTIQC                                                      -
-[cdcgov/phoenix] Pipeline completed with errors-
WARN: To render the execution DAG in the required format it is required to install Graphviz -- See http://www.graphviz.org for more info.
Error executing process > 'PHOENIX:PHOENIX_EXTERNAL:AMRFINDERPLUS_RUN (AR-0074)'

Caused by:
  Process `PHOENIX:PHOENIX_EXTERNAL:AMRFINDERPLUS_RUN (AR-0074)` terminated with an error exit status (1)

Command executed:

  #adding python path for running srst2 on terra
  
  
  if [[ AR-0074.filtered.scaffolds.fa.gz = *.gz ]]; then
      NUC_FNAME=$(basename AR-0074.filtered.scaffolds.fa.gz .gz)
      gzip -c -d AR-0074.filtered.scaffolds.fa.gz > $NUC_FNAME
  else
      NUC_FNAME = AR-0074.filtered.scaffolds.fa.gz
  fi
  
  # decompress the amrfinder database
  tar xzvf amrfinderdb_v3.11_20231115.1.tar.gz
  
  amrfinder \
      --nucleotide $NUC_FNAME \
      --protein AR-0074.faa \
      --gff AR-0074.gff \
      --annotation_format prokka \
      --mutation_all AR-0074_all_mutations.tsv \
      --organism Klebsiella_oxytoca \
      --plus \
      --database amrfinderdb_v3.11_20231115.1 \
      --threads 6 > AR-0074_all_genes.tsv
  
  sed -i '1s/ /_/g' AR-0074_all_genes.tsv
  
  cat <<-END_VERSIONS > versions.yml
  "PHOENIX:PHOENIX_EXTERNAL:AMRFINDERPLUS_RUN":
      amrfinderplus: $(amrfinder --version)
      amrfinderplus_db_version: $(head amrfinderdb_v3.11_20231115.1/version.txt)
      amrfinderplus_container: sha256:2311400ff8576f04e77a250e0665daa05f11b67f6a5901c8eac267378dccc932 
  END_VERSIONS
  
  #revert python path back to main envs for running on terra

Command exit status:
  1

Command output:
  amrfinderdb_v3.11_20231115.1/AMR_CDS.ndb
  amrfinderdb_v3.11_20231115.1/AMRProt.ptf
  amrfinderdb_v3.11_20231115.1/AMRProt.psq
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Escherichia.nto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecium.tab
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Streptococcus_pneumoniae.nto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Salmonella.njs
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Klebsiella_oxytoca.tab
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Salmonella.nto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Campylobacter.not
  amrfinderdb_v3.11_20231115.1/AMRProt
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Acinetobacter_baumannii.nhr
  amrfinderdb_v3.11_20231115.1/AMRProt.phr
  amrfinderdb_v3.11_20231115.1/AMR.LIB.h3p
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Escherichia.njs
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecalis.nhr
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Campylobacter.nto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Clostridioides_difficile.nsq
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecium.nhr
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Clostridioides_difficile.not
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Escherichia.ntf
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Neisseria_gonorrhoeae.njs
  amrfinderdb_v3.11_20231115.1/AMR_CDS.nsq
  amrfinderdb_v3.11_20231115.1/AMRProt.pto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Streptococcus_pneumoniae.ndb
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecium.nin
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecium.nsq
  amrfinderdb_v3.11_20231115.1/ReferenceGeneCatalog.txt
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecium.ntf
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Staphylococcus_aureus.tab
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecium.nto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Neisseria_gonorrhoeae.tab
  amrfinderdb_v3.11_20231115.1/AMRProt.pjs
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Campylobacter.ndb
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Campylobacter.tab
  amrfinderdb_v3.11_20231115.1/AMR_CDS.nto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Staphylococcus_aureus.not
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecalis.ndb
  amrfinderdb_v3.11_20231115.1/AMRProt.pin
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Klebsiella_oxytoca.nhr
  amrfinderdb_v3.11_20231115.1/AMR_CDS.nhr
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Streptococcus_pneumoniae.njs
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Acinetobacter_baumannii.nin
  amrfinderdb_v3.11_20231115.1/AMR.LIB.h3m
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Clostridioides_difficile.tab
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Clostridioides_difficile.nhr
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Streptococcus_pneumoniae.nin
  amrfinderdb_v3.11_20231115.1/AMR.LIB
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Clostridioides_difficile.ndb
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Neisseria_gonorrhoeae.nsq

Command error:
  Running: amrfinder --nucleotide AR-0074.filtered.scaffolds.fa --protein AR-0074.faa --gff AR-0074.gff --annotation_format prokka --mutation_all AR-0074_all_mutations.tsv --organism Klebsiella_oxytoca --plus --database amrfinderdb_v3.11_20231115.1 --threads 6
  Software directory: '/amrfinder/'
  Software version: 3.11.26
  Database directory: 'amrfinderdb_v3.11_20231115.1'
  Database version: 2023-11-15.1
  AMRFinder combined translated and protein and mutation search
  Running blastp
  Running hmmsearch
  Running tblastn
  Running blastn
  
  *** ERROR ***
  '/ncbi-blast-2.14.0+/bin/blastn'  -query 'AR-0074.filtered.scaffolds.fa' -db /tmp/amrfinder.tXGtz7/db/AMR_DNA-Klebsiella_oxytoca -evalue 1e-20  -dust no  -max_target_seqs 10000    -num_threads 2  -mt_mode 1 -outfmt '6 qseqid sseqid qstart qend qlen sstart send slen qseq sseq' -out /tmp/amrfinder.tXGtz7/blastn > /tmp/amrfinder.tXGtz7/log 2> /tmp/amrfinder.tXGtz7/blastn-err
  status = 35584
  Segmentation fault (core dumped)
  
  HOSTNAME: 1dcd48de1792
  SHELL: ?
  PWD: /scratch/zl7w2/tools/phoenix/assets/databases/work/66/48cce13659da4d61b03151dcf8ebc0
  PATH: /home/zl7w2/.nextflow/assets/cdcgov/phoenix/bin:/amrfinder:/ncbi-blast-2.14.0+/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  Progam name:  amrfinder
  Command line: amrfinder --nucleotide AR-0074.filtered.scaffolds.fa --protein AR-0074.faa --gff AR-0074.gff --annotation_format prokka --mutation_all AR-0074_all_mutations.tsv --organism Klebsiella_oxytoca --plus --database amrfinderdb_v3.11_20231115.1 --threads 6

Work dir:
  /scratch/zl7w2/tools/phoenix/assets/databases/work/66/48cce13659da4d61b03151dcf8ebc0

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

@jvhagey
Copy link
Contributor

jvhagey commented Mar 12, 2024

Is there anything else in the .command.err or .command.out files that are in the work dir listed? Is this a public sample I could try and run on my end to see if I can reproduce the problem?

@tang27abu
Copy link

tang27abu commented Mar 12, 2024 •
edited
Loading

@tang27abu, can you provide more detail on this sample? Is it an AR Bank sample?

Here is the command: nextflow run cdcgov/phoenix -r v2.1.1-dev -latest -profile singularity -entry PHOENIX --input samplesheet.csv --kraken2db $kraken2db

@tang27abu, can you provide more detail on this sample? Is it an AR Bank sample?
Here is the command: nextflow run cdcgov/phoenix -r v2.1.1-dev -latest -profile singularity -entry PHOENIX --input samplesheet.csv --kraken2db $kraken2db

I was trying to run with this command nextflow run cdcgov/phoenix -r v2.1.1-dev /scratch/zl7w2/tools/phoenix/main.nf -profile docker -entry PHOENIX --input /scratch/zl7w2/Matt/MUCRO4/reads/metadata.csv --kraken2db /scratch/zl7w2/tools/phoenix/k2db And get the error below:

[50/44fca7] process > PHOENIX:PHOENIX_EXTERNAL:GET_TAXA_FOR_AMRFINDER (AR-0074)                             [100%] 4 of 4
[66/48cce1] process > PHOENIX:PHOENIX_EXTERNAL:AMRFINDERPLUS_RUN (AR-0074)                                  [100%] 4 of 4, failed: 1
[9d/a4abd0] process > PHOENIX:PHOENIX_EXTERNAL:CALCULATE_ASSEMBLY_RATIO (AR-0074)                           [100%] 4 of 4
[e6/2af222] process > PHOENIX:PHOENIX_EXTERNAL:GENERATE_PIPELINE_STATS_WF:GENERATE_PIPELINE_STATS (AR-0034) [100%] 3 of 3
[f2/01cce4] process > PHOENIX:PHOENIX_EXTERNAL:CREATE_SUMMARY_LINE (AR-0034)                                [100%] 3 of 3
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:FETCH_FAILED_SUMMARIES                                       -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:GATHER_SUMMARY_LINES                                         -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:GRIPHIN                                                      -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:CUSTOM_DUMPSOFTWAREVERSIONS                                  -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:MULTIQC                                                      -
-[cdcgov/phoenix] Pipeline completed with errors-
WARN: To render the execution DAG in the required format it is required to install Graphviz -- See http://www.graphviz.org for more info.
Error executing process > 'PHOENIX:PHOENIX_EXTERNAL:AMRFINDERPLUS_RUN (AR-0074)'

Caused by:
  Process `PHOENIX:PHOENIX_EXTERNAL:AMRFINDERPLUS_RUN (AR-0074)` terminated with an error exit status (1)

Command executed:

  #adding python path for running srst2 on terra
  
  
  if [[ AR-0074.filtered.scaffolds.fa.gz = *.gz ]]; then
      NUC_FNAME=$(basename AR-0074.filtered.scaffolds.fa.gz .gz)
      gzip -c -d AR-0074.filtered.scaffolds.fa.gz > $NUC_FNAME
  else
      NUC_FNAME = AR-0074.filtered.scaffolds.fa.gz
  fi
  
  # decompress the amrfinder database
  tar xzvf amrfinderdb_v3.11_20231115.1.tar.gz
  
  amrfinder \
      --nucleotide $NUC_FNAME \
      --protein AR-0074.faa \
      --gff AR-0074.gff \
      --annotation_format prokka \
      --mutation_all AR-0074_all_mutations.tsv \
      --organism Klebsiella_oxytoca \
      --plus \
      --database amrfinderdb_v3.11_20231115.1 \
      --threads 6 > AR-0074_all_genes.tsv
  
  sed -i '1s/ /_/g' AR-0074_all_genes.tsv
  
  cat <<-END_VERSIONS > versions.yml
  "PHOENIX:PHOENIX_EXTERNAL:AMRFINDERPLUS_RUN":
      amrfinderplus: $(amrfinder --version)
      amrfinderplus_db_version: $(head amrfinderdb_v3.11_20231115.1/version.txt)
      amrfinderplus_container: sha256:2311400ff8576f04e77a250e0665daa05f11b67f6a5901c8eac267378dccc932 
  END_VERSIONS
  
  #revert python path back to main envs for running on terra

Command exit status:
  1

Command output:
  amrfinderdb_v3.11_20231115.1/AMR_CDS.ndb
  amrfinderdb_v3.11_20231115.1/AMRProt.ptf
  amrfinderdb_v3.11_20231115.1/AMRProt.psq
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Escherichia.nto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecium.tab
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Streptococcus_pneumoniae.nto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Salmonella.njs
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Klebsiella_oxytoca.tab
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Salmonella.nto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Campylobacter.not
  amrfinderdb_v3.11_20231115.1/AMRProt
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Acinetobacter_baumannii.nhr
  amrfinderdb_v3.11_20231115.1/AMRProt.phr
  amrfinderdb_v3.11_20231115.1/AMR.LIB.h3p
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Escherichia.njs
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecalis.nhr
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Campylobacter.nto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Clostridioides_difficile.nsq
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecium.nhr
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Clostridioides_difficile.not
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Escherichia.ntf
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Neisseria_gonorrhoeae.njs
  amrfinderdb_v3.11_20231115.1/AMR_CDS.nsq
  amrfinderdb_v3.11_20231115.1/AMRProt.pto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Streptococcus_pneumoniae.ndb
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecium.nin
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecium.nsq
  amrfinderdb_v3.11_20231115.1/ReferenceGeneCatalog.txt
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecium.ntf
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Staphylococcus_aureus.tab
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecium.nto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Neisseria_gonorrhoeae.tab
  amrfinderdb_v3.11_20231115.1/AMRProt.pjs
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Campylobacter.ndb
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Campylobacter.tab
  amrfinderdb_v3.11_20231115.1/AMR_CDS.nto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Staphylococcus_aureus.not
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecalis.ndb
  amrfinderdb_v3.11_20231115.1/AMRProt.pin
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Klebsiella_oxytoca.nhr
  amrfinderdb_v3.11_20231115.1/AMR_CDS.nhr
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Streptococcus_pneumoniae.njs
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Acinetobacter_baumannii.nin
  amrfinderdb_v3.11_20231115.1/AMR.LIB.h3m
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Clostridioides_difficile.tab
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Clostridioides_difficile.nhr
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Streptococcus_pneumoniae.nin
  amrfinderdb_v3.11_20231115.1/AMR.LIB
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Clostridioides_difficile.ndb
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Neisseria_gonorrhoeae.nsq

Command error:
  Running: amrfinder --nucleotide AR-0074.filtered.scaffolds.fa --protein AR-0074.faa --gff AR-0074.gff --annotation_format prokka --mutation_all AR-0074_all_mutations.tsv --organism Klebsiella_oxytoca --plus --database amrfinderdb_v3.11_20231115.1 --threads 6
  Software directory: '/amrfinder/'
  Software version: 3.11.26
  Database directory: 'amrfinderdb_v3.11_20231115.1'
  Database version: 2023-11-15.1
  AMRFinder combined translated and protein and mutation search
  Running blastp
  Running hmmsearch
  Running tblastn
  Running blastn
  
  *** ERROR ***
  '/ncbi-blast-2.14.0+/bin/blastn'  -query 'AR-0074.filtered.scaffolds.fa' -db /tmp/amrfinder.tXGtz7/db/AMR_DNA-Klebsiella_oxytoca -evalue 1e-20  -dust no  -max_target_seqs 10000    -num_threads 2  -mt_mode 1 -outfmt '6 qseqid sseqid qstart qend qlen sstart send slen qseq sseq' -out /tmp/amrfinder.tXGtz7/blastn > /tmp/amrfinder.tXGtz7/log 2> /tmp/amrfinder.tXGtz7/blastn-err
  status = 35584
  Segmentation fault (core dumped)
  
  HOSTNAME: 1dcd48de1792
  SHELL: ?
  PWD: /scratch/zl7w2/tools/phoenix/assets/databases/work/66/48cce13659da4d61b03151dcf8ebc0
  PATH: /home/zl7w2/.nextflow/assets/cdcgov/phoenix/bin:/amrfinder:/ncbi-blast-2.14.0+/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  Progam name:  amrfinder
  Command line: amrfinder --nucleotide AR-0074.filtered.scaffolds.fa --protein AR-0074.faa --gff AR-0074.gff --annotation_format prokka --mutation_all AR-0074_all_mutations.tsv --organism Klebsiella_oxytoca --plus --database amrfinderdb_v3.11_20231115.1 --threads 6

Work dir:
  /scratch/zl7w2/tools/phoenix/assets/databases/work/66/48cce13659da4d61b03151dcf8ebc0

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

Is there anything else in the .command.err or .command.out files that are in the work dir listed? Is this a public sample I could try and run on my end to see if I can reproduce the problem?

The .command.err and command.out are empty. I am starting another run to see if I can get any information. Here is the sample info

ID Organism Biosample
32 Enterobacter cloacae SAMN04014873
34 Klebsiella pneumoniae SAMN04014875
74 Klebsiella aerogenes SAMN04014915
76 Klebsiella pneumoniae SAMN04014917
93 Enterobacter cloacae SAMN04014917
100 Pseudomonas aeruginosa SAMN04014941

@tang27abu
Copy link

Hi,

The new run get this error, not sure why it cannot find the ResGANNCBI_20240131_srst2.fasta

[90/25d10f] process > PHOENIX:PHOENIX_EXTERNAL:GAMMA_AR (AR-0093)                                 [100%] 1 of 1, failed: 1
[bf/b540c6] process > PHOENIX:PHOENIX_EXTERNAL:GAMMA_PF (AR-0093)                                 [100%] 1 of 1 ✔
[b6/a23a64] process > PHOENIX:PHOENIX_EXTERNAL:QUAST (AR-0093)                                    [100%] 1 of 1 ✔
[22/2d8e71] process > PHOENIX:PHOENIX_EXTERNAL:KRAKEN2_WTASMBLD:KRAKEN2_WTASMBLD (AR-0093)        [100%] 1 of 1
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:KRAKEN2_WTASMBLD:KRAKENTOOLS_MAKEKREPORT           -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:KRAKEN2_WTASMBLD:KREPORT2KRONA_WTASMBLD            -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:KRAKEN2_WTASMBLD:KRAKEN2_BH_WTASMBLD               -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:KRAKEN2_WTASMBLD:KRONA_KTIMPORTTEXT_WTASMBLD       -
[d9/95c122] process > PHOENIX:PHOENIX_EXTERNAL:MASH_DIST (AR-0093)                                [100%] 1 of 1 ✔
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:DETERMINE_TOP_MASH_HITS                            -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:FASTANI                                            -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:FORMAT_ANI                                         -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:DETERMINE_TAXA_ID                                  -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:DO_MLST:MLST                                       -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:DO_MLST:CHECK_MLST                                 -
[4c/2411eb] process > PHOENIX:PHOENIX_EXTERNAL:PROKKA (AR-0093)                                   [100%] 1 of 1 ✔
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:GET_TAXA_FOR_AMRFINDER                             -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:AMRFINDERPLUS_RUN                                  -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:CALCULATE_ASSEMBLY_RATIO                           -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:GENERATE_PIPELINE_STATS_WF:GENERATE_PIPELINE_STATS -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:CREATE_SUMMARY_LINE                                -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:FETCH_FAILED_SUMMARIES                             -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:GATHER_SUMMARY_LINES                               -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:GRIPHIN                                            -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:CUSTOM_DUMPSOFTWAREVERSIONS                        -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:MULTIQC                                            -
-[cdcgov/phoenix] Pipeline completed with errors-
WARN: To render the execution DAG in the required format it is required to install Graphviz -- See http://www.graphviz.org for more info.
Error executing process > 'PHOENIX:PHOENIX_EXTERNAL:GAMMA_AR (AR-0093)'

Caused by:
  Process `PHOENIX:PHOENIX_EXTERNAL:GAMMA_AR (AR-0093)` terminated with an error exit status (1)

Command executed:

  db_name=$(echo ResGANNCBI_20240131_srst2.fasta | sed 's:.*/::' | sed 's/.fasta//')
  if [[ AR-0093.filtered.scaffolds.fa.gz == *.gz ]]
  then
      FNAME=$(basename AR-0093.filtered.scaffolds.fa.gz .gz)
      gunzip -f AR-0093.filtered.scaffolds.fa.gz
      GAMMA.py \
       \
      $FNAME \
      ResGANNCBI_20240131_srst2.fasta \
      AR-0093_$db_name
  else
      GAMMA.py \
       \
      AR-0093.filtered.scaffolds.fa.gz \
      ResGANNCBI_20240131_srst2.fasta \
      AR-0093_$db_name
  fi
  
  cat <<-END_VERSIONS > versions.yml
  "PHOENIX:PHOENIX_EXTERNAL:GAMMA_AR":
      gamma: 2.2
      gamma_container: sha256:60d8ac58e016349a856fb7b443dd422ba69bae3f40e0dad83460d25ecf71101e
      Database: ResGANNCBI_20240131_srst2.fasta
  END_VERSIONS

Command exit status:
  1

Command output:
  (empty)

Command error:
  mustOpen: Can't open ResGANNCBI_20240131_srst2.fasta to read: No such file or directory
  Traceback (most recent call last):
    File "/GAMMA/GAMMA.py", line 1067, in <module>
      sys.exit(main())
    File "/GAMMA/GAMMA.py", line 1056, in main
      GAMMA_Out(args.input_fasta, args.database, args.output,  args.percent_identity, Verbose)
    File "/GAMMA/GAMMA.py", line 1004, in GAMMA_Out
      GAMA_Output(output +'.psl', fasta, gene_db, output + '.gamma', verbose)
    File "/GAMMA/GAMMA.py", line 932, in GAMA_Output
      List1 = GAMA_List(PSL, genome_fasta, genes_fasta, verbose)
    File "/GAMMA/GAMMA.py", line 922, in GAMA_List
      Lines = GAMA_Line_Maker(PSL, genome_fasta, genes_fasta, verbose)
    File "/GAMMA/GAMMA.py", line 729, in GAMA_Line_Maker
      Genes = SeqIO.to_dict(SeqIO.parse(genes_fasta, 'fasta'))
    File "/usr/local/lib/python3.6/dist-packages/Bio/SeqIO/__init__.py", line 605, in parse
      return iterator_generator(handle)
    File "/usr/local/lib/python3.6/dist-packages/Bio/SeqIO/FastaIO.py", line 183, in __init__
      super().__init__(source, mode="t", fmt="Fasta")
    File "/usr/local/lib/python3.6/dist-packages/Bio/SeqIO/Interfaces.py", line 48, in __init__
      self.stream = open(source, "r" + mode)
  FileNotFoundError: [Errno 2] No such file or directory: 'ResGANNCBI_20240131_srst2.fasta'

Work dir:
  /scratch/zl7w2/tools/phoenix/work/90/25d10f652e53eb05bdff86202b03ba

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

This the the file list under work dir
Screenshot 2024-03-12 at 3 37 29 PM

@jvhagey
Copy link
Contributor

jvhagey commented Mar 12, 2024 •
edited
Loading

@tang27abu something is off here remove this from your command /scratch/zl7w2/tools/phoenix/main.nf so the it should just be nextflow run cdcgov/phoenix -r v2.1.1-dev -latest -profile docker -entry PHOENIX --input /scratch/zl7w2/Matt/MUCRO4/reads/metadata.csv --kraken2db /scratch/zl7w2/tools/phoenix/k2db

Can you tell me about the system you are running on? If its a laptop/desktop do you have other programs running on it at the same time?

@tang27abu
Copy link

@tang27abu something is off here remove this from your command /scratch/zl7w2/tools/phoenix/main.nf so the it should just be nextflow run cdcgov/phoenix -r v2.1.1-dev -profile docker -entry PHOENIX --input /scratch/zl7w2/Matt/MUCRO4/reads/metadata.csv --kraken2db /scratch/zl7w2/tools/phoenix/k2db

Can you tell me about the system you are running on? If its a laptop/desktop do you have other programs running on it at the same time?

I am running on a Ubuntu system.

@tang27abu
Copy link

@jvhagey
I run the command nextflow run cdcgov/phoenix -r v2.1.1-dev -profile docker -entry PHOENIX --input /scratch/zl7w2/Matt/MUCRO4/reads/metadata.csv --kraken2db /scratch/zl7w2/tools/phoenix/k2db

below are the error, still cannot find that file.

[-        ] process > PHOENIX:PHOENIX_EXTERNAL:SPADES_WF:GENERATE_PIPELINE_STATS_FAILURE          -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:SPADES_WF:CREATE_SUMMARY_LINE_FAILURE              -
[37/1f83d0] process > PHOENIX:PHOENIX_EXTERNAL:RENAME_FASTA_HEADERS (AR-0093)                     [100%] 1 of 1
[73/249f8b] process > PHOENIX:PHOENIX_EXTERNAL:BBMAP_REFORMAT (AR-0093)                           [100%] 1 of 1
[4e/7146fc] process > PHOENIX:PHOENIX_EXTERNAL:SCAFFOLD_COUNT_CHECK (AR-0093)                     [100%] 1 of 1
[d8/b9527b] process > PHOENIX:PHOENIX_EXTERNAL:GAMMA_HV (AR-0093)                                 [100%] 1 of 1
[17/2f4f24] process > PHOENIX:PHOENIX_EXTERNAL:GAMMA_AR (AR-0093)                                 [100%] 1 of 1, failed: 1
[0d/68e78d] process > PHOENIX:PHOENIX_EXTERNAL:GAMMA_PF (AR-0093)                                 [100%] 1 of 1 ✔
[4f/6c6205] process > PHOENIX:PHOENIX_EXTERNAL:QUAST (AR-0093)                                    [100%] 1 of 1 ✔
[71/6c1238] process > PHOENIX:PHOENIX_EXTERNAL:KRAKEN2_WTASMBLD:KRAKEN2_WTASMBLD (AR-0093)        [100%] 1 of 1
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:KRAKEN2_WTASMBLD:KRAKENTOOLS_MAKEKREPORT           -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:KRAKEN2_WTASMBLD:KREPORT2KRONA_WTASMBLD            -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:KRAKEN2_WTASMBLD:KRAKEN2_BH_WTASMBLD               -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:KRAKEN2_WTASMBLD:KRONA_KTIMPORTTEXT_WTASMBLD       -
[51/af178c] process > PHOENIX:PHOENIX_EXTERNAL:MASH_DIST (AR-0093)                                [100%] 1 of 1 ✔
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:DETERMINE_TOP_MASH_HITS                            -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:FASTANI                                            -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:FORMAT_ANI                                         -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:DETERMINE_TAXA_ID                                  -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:DO_MLST:MLST                                       -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:DO_MLST:CHECK_MLST                                 -
[64/feb7ca] process > PHOENIX:PHOENIX_EXTERNAL:PROKKA (AR-0093)                                   [100%] 1 of 1 ✔
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:GET_TAXA_FOR_AMRFINDER                             -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:AMRFINDERPLUS_RUN                                  -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:CALCULATE_ASSEMBLY_RATIO                           -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:GENERATE_PIPELINE_STATS_WF:GENERATE_PIPELINE_STATS -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:CREATE_SUMMARY_LINE                                -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:FETCH_FAILED_SUMMARIES                             -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:GATHER_SUMMARY_LINES                               -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:GRIPHIN                                            -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:CUSTOM_DUMPSOFTWAREVERSIONS                        -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:MULTIQC                                            -
-[cdcgov/phoenix] Pipeline completed with errors-
WARN: To render the execution DAG in the required format it is required to install Graphviz -- See http://www.graphviz.org for more info.
Error executing process > 'PHOENIX:PHOENIX_EXTERNAL:GAMMA_AR (AR-0093)'

Caused by:
  Process `PHOENIX:PHOENIX_EXTERNAL:GAMMA_AR (AR-0093)` terminated with an error exit status (1)

Command executed:

  db_name=$(echo ResGANNCBI_20240131_srst2.fasta | sed 's:.*/::' | sed 's/.fasta//')
  if [[ AR-0093.filtered.scaffolds.fa.gz == *.gz ]]
  then
      FNAME=$(basename AR-0093.filtered.scaffolds.fa.gz .gz)
      gunzip -f AR-0093.filtered.scaffolds.fa.gz
      GAMMA.py \
       \
      $FNAME \
      ResGANNCBI_20240131_srst2.fasta \
      AR-0093_$db_name
  else
      GAMMA.py \
       \
      AR-0093.filtered.scaffolds.fa.gz \
      ResGANNCBI_20240131_srst2.fasta \
      AR-0093_$db_name
  fi
  
  cat <<-END_VERSIONS > versions.yml
  "PHOENIX:PHOENIX_EXTERNAL:GAMMA_AR":
      gamma: 2.2
      gamma_container: sha256:60d8ac58e016349a856fb7b443dd422ba69bae3f40e0dad83460d25ecf71101e
      Database: ResGANNCBI_20240131_srst2.fasta
  END_VERSIONS

Command exit status:
  1

Command output:
  (empty)

Command error:
  mustOpen: Can't open ResGANNCBI_20240131_srst2.fasta to read: No such file or directory
  Traceback (most recent call last):
    File "/GAMMA/GAMMA.py", line 1067, in <module>
      sys.exit(main())
    File "/GAMMA/GAMMA.py", line 1056, in main
      GAMMA_Out(args.input_fasta, args.database, args.output,  args.percent_identity, Verbose)
    File "/GAMMA/GAMMA.py", line 1004, in GAMMA_Out
      GAMA_Output(output +'.psl', fasta, gene_db, output + '.gamma', verbose)
    File "/GAMMA/GAMMA.py", line 932, in GAMA_Output
      List1 = GAMA_List(PSL, genome_fasta, genes_fasta, verbose)
    File "/GAMMA/GAMMA.py", line 922, in GAMA_List
      Lines = GAMA_Line_Maker(PSL, genome_fasta, genes_fasta, verbose)
    File "/GAMMA/GAMMA.py", line 729, in GAMA_Line_Maker
      Genes = SeqIO.to_dict(SeqIO.parse(genes_fasta, 'fasta'))
    File "/usr/local/lib/python3.6/dist-packages/Bio/SeqIO/__init__.py", line 605, in parse
      return iterator_generator(handle)
    File "/usr/local/lib/python3.6/dist-packages/Bio/SeqIO/FastaIO.py", line 183, in __init__
      super().__init__(source, mode="t", fmt="Fasta")
    File "/usr/local/lib/python3.6/dist-packages/Bio/SeqIO/Interfaces.py", line 48, in __init__
      self.stream = open(source, "r" + mode)
  FileNotFoundError: [Errno 2] No such file or directory: 'ResGANNCBI_20240131_srst2.fasta'

Work dir:
  /scratch/zl7w2/tools/phoenix/work/17/2f4f24d32e5c6008e492fded65d08d

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line
Screenshot 2024-03-12 at 4 51 16 PM

@tang27abu
Copy link

should I git pull the repo first then run that command?

@jvhagey
Copy link
Contributor

jvhagey commented Mar 13, 2024

@tang27abu anytime you are testing a dev branch you need to add -latest to your command. Dev branches are constantly in flux and I can't be sure we are comparing the same code unless you use -latest, which will automatically pull the latest code from github if your copy isn't up-to-date. This only works if you are using nextflow run cdcgov/phoenix and not running main.nf after cloning/pulling the repo yourself.

In terms of your system, can you let me know the specs? What size is your RAM? The original problem might still be a memory one if you are using a laptop and have other programs running at the same time.

@tang27abu
Copy link

@tang27abu anytime you are testing a dev branch you need to add -latest to your command. Dev branches are constantly in flux and I can't be sure we are comparing the same code unless you use -latest, which will automatically pull the latest code from github if your copy isn't up-to-date. This only works if you are using nextflow run cdcgov/phoenix and not running main.nf after cloning/pulling the repo yourself.

In terms of your system, can you let me know the specs? What size is your RAM? The original problem might still be a memory one if you are using a laptop and have other programs running at the same time.

I run this command nextflow run cdcgov/phoenix -r v2.1.1-dev -latest -profile docker -entry PHOENIX --input /scratch/zl7w2/Matt/MUCRO4/reads/metadata.csv --kraken2db /scratch/zl7w2/tools/phoenix/k2db but still get the same error, FileNotFoundError: [Errno 2] No such file or directory: 'ResGANNCBI_20240131_srst2.fasta'

Here is the memory informations:
Screenshot 2024-03-13 at 10 03 02 AM

@tang27abu
Copy link

When I remove the sample 0074 - SAMN04014915, it works well with ver2.1.0.

@jvhagey
Copy link
Contributor

jvhagey commented Mar 13, 2024

Does the sample run on its own? Like if you didn't have it with other samples going at the same time?

@tang27abu
Copy link

Does the sample run on its own? Like if you didn't have it with other samples going at the same time?

Just try only run with 0074 sample and it gives same core dumped error

[ba/5836ea] process > PHOENIX:PHOENIX_EXTERNAL:GET_TAXA_FOR_AMRFINDER (AR-0074)                       [100%] 1 of 1 ✔
[7e/495ec8] process > PHOENIX:PHOENIX_EXTERNAL:AMRFINDERPLUS_RUN (AR-0074)                            [100%] 1 of 1, failed: 1 ✘
[02/cb9c30] process > PHOENIX:PHOENIX_EXTERNAL:CALCULATE_ASSEMBLY_RATIO (AR-0074)                     [100%] 1 of 1 ✔
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:GENERATE_PIPELINE_STATS_WF:GENERATE_PIPELINE_STATS     -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:CREATE_SUMMARY_LINE                                    -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:FETCH_FAILED_SUMMARIES (1)                             -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:GATHER_SUMMARY_LINES                                   -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:GRIPHIN                                                -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:CUSTOM_DUMPSOFTWAREVERSIONS                            -
[-        ] process > PHOENIX:PHOENIX_EXTERNAL:MULTIQC                                                -
Execution cancelled -- Finishing pending tasks before exit
-[cdcgov/phoenix] Pipeline completed with errors-
WARN: Killing running tasks (1)
WARN: Graphviz is required to render the execution DAG in the given format -- See http://www.graphviz.org for more info.
ERROR ~ Error executing process > 'PHOENIX:PHOENIX_EXTERNAL:AMRFINDERPLUS_RUN (AR-0074)'

Caused by:
  Process `PHOENIX:PHOENIX_EXTERNAL:AMRFINDERPLUS_RUN (AR-0074)` terminated with an error exit status (1)

Command executed:

  #adding python path for running srst2 on terra
  
  
  if [[ AR-0074.filtered.scaffolds.fa.gz = *.gz ]]; then
      NUC_FNAME=$(basename AR-0074.filtered.scaffolds.fa.gz .gz)
      gzip -c -d AR-0074.filtered.scaffolds.fa.gz > $NUC_FNAME
  else
      NUC_FNAME = AR-0074.filtered.scaffolds.fa.gz
  fi
  
  # decompress the amrfinder database
  tar xzvf amrfinderdb_v3.11_20231115.1.tar.gz
  
  amrfinder \
      --nucleotide $NUC_FNAME \
      --protein AR-0074.faa \
      --gff AR-0074.gff \
      --annotation_format prokka \
      --mutation_all AR-0074_all_mutations.tsv \
      --organism Klebsiella_oxytoca \
      --plus \
      --database amrfinderdb_v3.11_20231115.1 \
      --threads 6 > AR-0074_all_genes.tsv
  
  sed -i '1s/ /_/g' AR-0074_all_genes.tsv
  
  cat <<-END_VERSIONS > versions.yml
  "PHOENIX:PHOENIX_EXTERNAL:AMRFINDERPLUS_RUN":
      amrfinderplus: $(amrfinder --version)
      amrfinderplus_db_version: $(head amrfinderdb_v3.11_20231115.1/version.txt)
      amrfinderplus_container: sha256:2311400ff8576f04e77a250e0665daa05f11b67f6a5901c8eac267378dccc932 
  END_VERSIONS
  
  #revert python path back to main envs for running on terra

Command exit status:
  1

Command output:
  amrfinderdb_v3.11_20231115.1/AMR_CDS.ndb
  amrfinderdb_v3.11_20231115.1/AMRProt.ptf
  amrfinderdb_v3.11_20231115.1/AMRProt.psq
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Escherichia.nto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecium.tab
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Streptococcus_pneumoniae.nto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Salmonella.njs
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Klebsiella_oxytoca.tab
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Salmonella.nto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Campylobacter.not
  amrfinderdb_v3.11_20231115.1/AMRProt
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Acinetobacter_baumannii.nhr
  amrfinderdb_v3.11_20231115.1/AMRProt.phr
  amrfinderdb_v3.11_20231115.1/AMR.LIB.h3p
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Escherichia.njs
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecalis.nhr
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Campylobacter.nto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Clostridioides_difficile.nsq
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecium.nhr
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Clostridioides_difficile.not
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Escherichia.ntf
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Neisseria_gonorrhoeae.njs
  amrfinderdb_v3.11_20231115.1/AMR_CDS.nsq
  amrfinderdb_v3.11_20231115.1/AMRProt.pto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Streptococcus_pneumoniae.ndb
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecium.nin
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecium.nsq
  amrfinderdb_v3.11_20231115.1/ReferenceGeneCatalog.txt
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecium.ntf
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Staphylococcus_aureus.tab
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecium.nto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Neisseria_gonorrhoeae.tab
  amrfinderdb_v3.11_20231115.1/AMRProt.pjs
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Campylobacter.ndb
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Campylobacter.tab
  amrfinderdb_v3.11_20231115.1/AMR_CDS.nto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Staphylococcus_aureus.not
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecalis.ndb
  amrfinderdb_v3.11_20231115.1/AMRProt.pin
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Klebsiella_oxytoca.nhr
  amrfinderdb_v3.11_20231115.1/AMR_CDS.nhr
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Streptococcus_pneumoniae.njs
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Acinetobacter_baumannii.nin
  amrfinderdb_v3.11_20231115.1/AMR.LIB.h3m
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Clostridioides_difficile.tab
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Clostridioides_difficile.nhr
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Streptococcus_pneumoniae.nin
  amrfinderdb_v3.11_20231115.1/AMR.LIB
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Clostridioides_difficile.ndb
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Neisseria_gonorrhoeae.nsq

Command error:
  amrfinderdb_v3.11_20231115.1/AMR_CDS.nsq
  amrfinderdb_v3.11_20231115.1/AMRProt.pto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Streptococcus_pneumoniae.ndb
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecium.nin
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecium.nsq
  amrfinderdb_v3.11_20231115.1/ReferenceGeneCatalog.txt
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecium.ntf
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Staphylococcus_aureus.tab
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecium.nto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Neisseria_gonorrhoeae.tab
  amrfinderdb_v3.11_20231115.1/AMRProt.pjs
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Campylobacter.ndb
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Campylobacter.tab
  amrfinderdb_v3.11_20231115.1/AMR_CDS.nto
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Staphylococcus_aureus.not
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Enterococcus_faecalis.ndb
  amrfinderdb_v3.11_20231115.1/AMRProt.pin
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Klebsiella_oxytoca.nhr
  amrfinderdb_v3.11_20231115.1/AMR_CDS.nhr
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Streptococcus_pneumoniae.njs
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Acinetobacter_baumannii.nin
  amrfinderdb_v3.11_20231115.1/AMR.LIB.h3m
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Clostridioides_difficile.tab
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Clostridioides_difficile.nhr
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Streptococcus_pneumoniae.nin
  amrfinderdb_v3.11_20231115.1/AMR.LIB
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Clostridioides_difficile.ndb
  amrfinderdb_v3.11_20231115.1/AMR_DNA-Neisseria_gonorrhoeae.nsq
  Running: amrfinder --nucleotide AR-0074.filtered.scaffolds.fa --protein AR-0074.faa --gff AR-0074.gff --annotation_format prokka --mutation_all AR-0074_all_mutations.tsv --organism Klebsiella_oxytoca --plus --database amrfinderdb_v3.11_20231115.1 --threads 6
  Software directory: '/amrfinder/'
  Software version: 3.11.26
  Database directory: 'amrfinderdb_v3.11_20231115.1'
  Database version: 2023-11-15.1
  AMRFinder combined translated and protein and mutation search
  Running blastp
  Running hmmsearch
  Running tblastn
  Running blastn
  
  *** ERROR ***
  '/ncbi-blast-2.14.0+/bin/blastn'  -query 'AR-0074.filtered.scaffolds.fa' -db /tmp/amrfinder.qJQEm6/db/AMR_DNA-Klebsiella_oxytoca -evalue 1e-20  -dust no  -max_target_seqs 10000    -num_threads 2  -mt_mode 1 -outfmt '6 qseqid sseqid qstart qend qlen sstart send slen qseq sseq' -out /tmp/amrfinder.qJQEm6/blastn > /tmp/amrfinder.qJQEm6/log 2> /tmp/amrfinder.qJQEm6/blastn-err
  status = 35584
  Segmentation fault (core dumped)
  
  HOSTNAME: 19244c2d84a4
  SHELL: ?
  PWD: /scratch/zl7w2/tools/phoenix_v2.0.0/work/7e/495ec8c96d461971049e51c21118f4
  PATH: /amrfinder:/ncbi-blast-2.14.0+/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/scratch/zl7w2/tools/phoenix/bin
  Progam name:  amrfinder
  Command line: amrfinder --nucleotide AR-0074.filtered.scaffolds.fa --protein AR-0074.faa --gff AR-0074.gff --annotation_format prokka --mutation_all AR-0074_all_mutations.tsv --organism Klebsiella_oxytoca --plus --database amrfinderdb_v3.11_20231115.1 --threads 6

Work dir:
  /scratch/zl7w2/tools/phoenix_v2.0.0/work/7e/495ec8c96d461971049e51c21118f4

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details

@jvhagey jvhagey mentioned this issue Mar 19, 2024
Merged
9 tasks
@HAISeq HAISeq linked a pull request Mar 25, 2024 that will close this issue
v2.1.1 dev merge to main #146
Merged
@jvhagey jvhagey added Bug: System Specific Something isn't working, specific to certain systems and removed Can't reproduce labels Mar 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jvhagey jvhagey

Labels
Bug: System Specific Something isn't working, specific to certain systems
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

v2.1.1 dev merge to main CDCgov/phoenix
3 participants
@tang27abu @jvhagey @SeanSierra-Patev