[JOSS Review] Error with Linux Tests - init and also Freebayes

[gbouras13]

Re openjournals/joss-reviews#6062

I ran the same code for #15 on a Linux box (Ubuntu 20.04).

git clone https://github.com/kdm9/Acanthophis.git
mamba env create -f environment-setup.yml -n acanthophis-demo
conda activate acanthophis-demo
pip install -U acanthophis # v0.3.0 was installed

cd Acanthophis/tests
acanthophis-init .
snakemake --snakefile Snakefile.generate-rawdata -j 8 --use-conda
snakemake snakemake -j 8 --use-conda --conda-frontend mamba

Unlike #15 on the Mac, the command acanthophis-init . did not work as designed - it created a structure like:

config.yml
environment.yml
rawdata/
workflow/

with the Snakefile and rules subdirectory in the workflow directory. I'm not really sure why.

In any case, once I copied the Snakefile, rules directory and config.schema.yml file into the correct location, acanthophis did indeed run as expected (the required conda envs were created, almost every rule ran well).

Specifically I ran:

snakemake -j 8 --use-conda --conda-frontend mamba --keep-going

However, there was an error:

Error in rule freebayes:
    jobid: 277
    input: rawdata/reference/genome.fa, output/variants/raw_split/bwa~lambda~all_samples.bamlist, output/variants/raw_split/bwa~lambda~all_samples.fosn
    output: tmp/variants/raw_split/freebayes~bwa~lambda~all_samples/NC_001416:16385-32768.bcf
    log: output/variants/raw_split/freebayes~bwa~lambda~all_samples/NC_001416:16385-32768.bcf.log (check log file(s) for error details)
    conda-env: /data/Acanthophis/tests/l/.snakemake/conda/bd7a1f75aa6d5405df665da7e2521080_
    shell:
        ( freebayes   --theta 0.01   --ploidy 2   --use-best-n-alleles 4      --min-mapping-quality 30   --min-base-quality 15   --read-max-mismatch-fraction 0.1   --skip-coverage 4000   --prob-contamination 0.05   --strict-vcf   --region 'NC_001416:16385-32768'   --fasta-reference rawdata/reference/genome.fa   --bam-list output/variants/raw_split/bwa~lambda~all_samples.bamlist   --samples output/variants/raw_split/bwa~lambda~all_samples.fosn | bcftools view   -O b6   -o tmp/variants/raw_split/freebayes~bwa~lambda~all_samples/NC_001416:16385-32768.bcf ) >output/variants/raw_split/freebayes~bwa~lambda~all_samples/NC_001416:16385-32768.bcf.log 2>&1
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Tue Jan 16 02:32:33 2024]
Error in rule freebayes:
    jobid: 275
    input: rawdata/reference/genome.fa, output/variants/raw_split/bwa~lambda~all_samples.bamlist, output/variants/raw_split/bwa~lambda~all_samples.fosn
    output: tmp/variants/raw_split/freebayes~bwa~lambda~all_samples/NC_001416:1-16384.bcf
    log: output/variants/raw_split/freebayes~bwa~lambda~all_samples/NC_001416:1-16384.bcf.log (check log file(s) for error details)
    conda-env: /data/Acanthophis/tests/l/.snakemake/conda/bd7a1f75aa6d5405df665da7e2521080_
    shell:
        ( freebayes   --theta 0.01   --ploidy 2   --use-best-n-alleles 4      --min-mapping-quality 30   --min-base-quality 15   --read-max-mismatch-fraction 0.1   --skip-coverage 4000   --prob-contamination 0.05   --strict-vcf   --region 'NC_001416:1-16384'   --fasta-reference rawdata/reference/genome.fa   --bam-list output/variants/raw_split/bwa~lambda~all_samples.bamlist   --samples output/variants/raw_split/bwa~lambda~all_samples.fosn | bcftools view   -O b6   -o tmp/variants/raw_split/freebayes~bwa~lambda~all_samples/NC_001416:1-16384.bcf ) >output/variants/raw_split/freebayes~bwa~lambda~all_samples/NC_001416:1-16384.bcf.log 2>&1
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Exiting because a job execution failed. Look above for error message

When I examined the log output/variants/raw_split/freebayes~bwa~lambda~all_samples/NC_001416\:1-16384.bcf.log it said

Failed to read from standard input: unknown file type

If it helps, the remaining DAG jobs are:

job                   count
------------------  -------
all                       1
bcffilter                 2
bcfmerge                  1
bcfmerge2group            1
finalvariantidx           1
freebayes                 2
premergevariantidx        2
vcfstats                  1
total                    11

Select jobs to execute...

I've inspected the rule's input files, they look as expected (I assume).

George

[kdm9]

Many many thanks for your efforts in review George!!

I ran the same code for #15 on a Linux box (Ubuntu 20.04).

git clone https://github.com/kdm9/Acanthophis.git
mamba env create -f environment-setup.yml -n acanthophis-demo
conda activate acanthophis-demo
pip install -U acanthophis # v0.3.0 was installed

cd Acanthophis/tests
acanthophis-init .
snakemake --snakefile Snakefile.generate-rawdata -j 8 --use-conda
snakemake snakemake -j 8 --use-conda --conda-frontend mamba

Unlike #15 on the Mac, the command acanthophis-init . did not work as designed - it created a structure like:

config.yml
environment.yml
rawdata/
workflow/

with the Snakefile and rules subdirectory in the workflow directory. I'm not really sure why.

That is intentional, the latest snakemake guidelines suggest having the workflow files under ./workflow.

In any case, once I copied the Snakefile, rules directory and config.schema.yml file into the correct location, acanthophis did indeed run as expected (the required conda envs were created, almost every rule ran well).

Copying the snakefile around could break things, although if it ran at all snakemake seems to have automatically figured out what was going on.

Specifically I ran:

snakemake -j 8 --use-conda --conda-frontend mamba --keep-going

However, there was an error:

Error in rule freebayes:
    jobid: 277
    input: rawdata/reference/genome.fa, output/variants/raw_split/bwa~lambda~all_samples.bamlist, output/variants/raw_split/bwa~lambda~all_samples.fosn
    output: tmp/variants/raw_split/freebayes~bwa~lambda~all_samples/NC_001416:16385-32768.bcf
    log: output/variants/raw_split/freebayes~bwa~lambda~all_samples/NC_001416:16385-32768.bcf.log (check log file(s) for error details)
    conda-env: /data/Acanthophis/tests/l/.snakemake/conda/bd7a1f75aa6d5405df665da7e2521080_
    shell:
        ( freebayes   --theta 0.01   --ploidy 2   --use-best-n-alleles 4      --min-mapping-quality 30   --min-base-quality 15   --read-max-mismatch-fraction 0.1   --skip-coverage 4000   --prob-contamination 0.05   --strict-vcf   --region 'NC_001416:16385-32768'   --fasta-reference rawdata/reference/genome.fa   --bam-list output/variants/raw_split/bwa~lambda~all_samples.bamlist   --samples output/variants/raw_split/bwa~lambda~all_samples.fosn | bcftools view   -O b6   -o tmp/variants/raw_split/freebayes~bwa~lambda~all_samples/NC_001416:16385-32768.bcf ) >output/variants/raw_split/freebayes~bwa~lambda~all_samples/NC_001416:16385-32768.bcf.log 2>&1
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Tue Jan 16 02:32:33 2024]
Error in rule freebayes:
    jobid: 275
    input: rawdata/reference/genome.fa, output/variants/raw_split/bwa~lambda~all_samples.bamlist, output/variants/raw_split/bwa~lambda~all_samples.fosn
    output: tmp/variants/raw_split/freebayes~bwa~lambda~all_samples/NC_001416:1-16384.bcf
    log: output/variants/raw_split/freebayes~bwa~lambda~all_samples/NC_001416:1-16384.bcf.log (check log file(s) for error details)
    conda-env: /data/Acanthophis/tests/l/.snakemake/conda/bd7a1f75aa6d5405df665da7e2521080_
    shell:
        ( freebayes   --theta 0.01   --ploidy 2   --use-best-n-alleles 4      --min-mapping-quality 30   --min-base-quality 15   --read-max-mismatch-fraction 0.1   --skip-coverage 4000   --prob-contamination 0.05   --strict-vcf   --region 'NC_001416:1-16384'   --fasta-reference rawdata/reference/genome.fa   --bam-list output/variants/raw_split/bwa~lambda~all_samples.bamlist   --samples output/variants/raw_split/bwa~lambda~all_samples.fosn | bcftools view   -O b6   -o tmp/variants/raw_split/freebayes~bwa~lambda~all_samples/NC_001416:1-16384.bcf ) >output/variants/raw_split/freebayes~bwa~lambda~all_samples/NC_001416:1-16384.bcf.log 2>&1
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Exiting because a job execution failed. Look above for error message

When I examined the log output/variants/raw_split/freebayes~bwa~lambda~all_samples/NC_001416\:1-16384.bcf.log it said

Failed to read from standard input: unknown file type

I'm pretty sure that this is a bcftools message, indicating that freebayes did not produce any output. I've tried in a fresh docker image, and I can't reproduce this. are there any other details you can provide?

If it helps, the remaining DAG jobs are:

job                   count
------------------  -------
all                       1
bcffilter                 2
bcfmerge                  1
bcfmerge2group            1
finalvariantidx           1
freebayes                 2
premergevariantidx        2
vcfstats                  1
total                    11

Select jobs to execute...

I've inspected the rule's input files, they look as expected (I assume).

One debugging measure you could do is run:

conda activate /data/Acanthophis/tests/l/.snakemake/conda/bd7a1f75aa6d5405df665da7e2521080_
freebayes   --theta 0.01   --ploidy 2   --use-best-n-alleles 4      --min-mapping-quality 30  \
     --min-base-quality 15 --read-max-mismatch-fraction 0.1 --skip-coverage 4000 --prob-contamination 0.05  \
     --strict-vcf   --region 'NC_001416:1-16384'   --fasta-reference rawdata/reference/genome.fa  \
     --bam-list output/variants/raw_split/bwa~lambda~all_samples.bamlist   \
     --samples output/variants/raw_split/bwa~lambda~all_samples.fosn  |less

In other words, just the freebayes part of the call that failed. It would also be useful to confirm the version of freebayes in that conda env.

thanks again,
kevin

[gbouras13]

Hi Kevin,

I got a Illegal instruction (core dumped) error with Freebayes v1.3.7.

I actually have run into issues with Freebayes v1.3.7 in a program of mine, pypolca, so I tried v1.3.6

mamba install freebayes==1.3.6

after I did this, acanthophis finished with no error.

[gbouras13]

I've made a small commit and PR to fix this in varcall.yaml - this should solve the issue.

I'll finish the rest of my review tomorrow now.

George

[kdm9]

yeah, ok, so that's independent from acanthophis then. I'm assuming you're using an old server that recent conda packages don't support?

I'm closing this, since it seems unrelated to acanthophis, but please reopen if there are further issues :)

[gbouras13]

No @kdm9 I don't think that is an issue with my server (my server is very new) - the v1.3.7 version of freebayes on bioconda is the newest, so it will be installed by default for most people unless you set a dependency limit <1.3.7. It is an external issue to Acanthophis but it will otherwise still affect it.

George

[gbouras13]

freebayes/freebayes#776

It seems to be a common issue

[kdm9]

Ah, sorry, in that case my bad, and i've merged the PR to pin the version. Thanks!