Unable to run tests: missing 'populate_metadata' in module 'acanthophis'

[bricoletc]

(Part of review thread openjournals/joss-reviews#6062)

Hello! I tried running the tests for this repo by following the instructions here and found the following minor issues:

• The conda environment file is currently not conda.yml as per the README, but environment-setup.yml
• In that environment-setup.yml, installing msprime from pip failed - see attached log file
  failed_pip_install_msprime.txt. I found that simply installing msprime from conda-forge worked instead, as per instructions here.
  --> Update the .yml accordingly?
• The last part of the README in the tests directory should be updated to reflect the current status of Acanthophis - i.e.,
  using the instructions from here in the main README, or just link to that.
  Same for the comment '# once I put this on PyPI:' - I see acanthophis is on PyPI

And here is a more major issue:

• Running snakemake --snakefile Snakefile.generate-rawdata -j 8 --use-conda completed fine and produced the rawdata. However, for actually running acanthophis on the generated
  test dataset, snakemake --snakefile Snakefile -j 8 --use-conda, Snakefile and config.yml are missing.
  I saw that inside the script test.sh there are instructions to generate them, using acanthophis-init. Please either
  tell the user to run setup.sh and test.sh, or acanthophis-init, in the README code block.
• Having done as above, command snakemake --snakefile Snakefile -j 8 --use-conda gives the following error:

  AttributeError in file /home/adminbrice/Desktop/research/writing/reviewing_exernal/JOSS/Acanthophis/test_tool/Acanthophis/tests/Snakefile, line 4:
    module 'acanthophis' has no attribute 'populate_metadata'
  File "/home/adminbrice/Desktop/research/writing/reviewing_exernal/JOSS/Acanthophis/test_tool/Acanthophis/tests/Snakefile", line 4, in <module>

[abhishektiwari]

Using environment-setup.yml I could not run test as well.

SSH to container,

docker run -it --rm condaforge/miniforge3

Then run

git clone https://github.com/kdm9/Acanthophis.git

(acanthophis-tests) root@5110a2faf4b9:/Acanthophis/tests# mamba env update -f environment-setup.yml
conda-forge/linux-aarch64                                   Using cache
conda-forge/noarch                                          Using cache
bioconda/linux-aarch64                                      Using cache
bioconda/noarch                                             Using cache


Looking for: ["python[version='>=3.11']", "snakemake[version='>=8.2']", "snakemake-interface-executor-plugins[version='>=8.2']", 'tree', 'pip', 'numpy', 'wgsim', 'centrifuge-core', 'samtools', 'msprime', 'curl']


Could not solve for environment specs
The following packages are incompatible
├─ centrifuge-core does not exist (perhaps a typo or a missing channel);
├─ msprime does not exist (perhaps a typo or a missing channel);
├─ samtools does not exist (perhaps a typo or a missing channel);
└─ wgsim does not exist (perhaps a typo or a missing channel).

I tried other methods to install msprime but it does not work,

(acanthophis-tests) root@5110a2faf4b9:/Acanthophis/tests# conda install -c conda-forge msprime
Channels:
 - conda-forge
 - bioconda
Platform: linux-aarch64
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - msprime

[kdm9]

RE @bricoletc's original issue

(Part of review thread openjournals/joss-reviews#6062)

Hello! I tried running the tests for this repo by following the instructions here and found the following minor issues:

* The conda environment file is currently not `conda.yml` as per the README, but `environment-setup.yml`

Oops, fixed in the readme. In fact I completely rewrote the readme as there were a few other errors there too.

* In that `environment-setup.yml`, installing msprime from pip failed - see attached log file
  [failed_pip_install_msprime.txt](https://github.com/kdm9/Acanthophis/files/13512718/failed_pip_install_msprime.txt). I found that simply installing msprime from conda-forge worked instead, as per [instructions here](https://tskit.dev/msprime/docs/latest/installation.html).
  --> Update the .yml accordingly?

Fixed, thanks.

* The last part of the README in the `tests` directory should be updated to reflect the current status of Acanthophis - i.e.,
  using the instructions from [here](https://github.com/kdm9/Acanthophis#documentation) in the main README, or just link to that.
  Same for the comment '# once I put this on PyPI:' - I see acanthophis is on PyPI

See above, rewrote readme, correct now.

And here is a more major issue:

* Running `snakemake --snakefile Snakefile.generate-rawdata -j 8 --use-conda` completed fine and produced the `rawdata`. However, for actually running acanthophis on the generated
  test dataset, `snakemake --snakefile Snakefile -j 8 --use-conda`, `Snakefile` and `config.yml` are missing.
  I saw that inside the script `test.sh` there are instructions to generate them, using `acanthophis-init`. Please either
  tell the user to run `setup.sh` and `test.sh`, or `acanthophis-init`, in the README code block.

Likewise, hopefully much clearer in the current readme.

* Having done as above, command `snakemake --snakefile Snakefile -j 8 --use-conda` gives the following error:
  ```shell
  AttributeError in file /home/adminbrice/Desktop/research/writing/reviewing_exernal/JOSS/Acanthophis/test_tool/Acanthophis/tests/Snakefile, line 4:
    module 'acanthophis' has no attribute 'populate_metadata'
  File "/home/adminbrice/Desktop/research/writing/reviewing_exernal/JOSS/Acanthophis/test_tool/Acanthophis/tests/Snakefile", line 4, in <module>
  ```

Can you confirm that the latest main branch has this issue still? I think it was fixed as part of one of the other issues you and @gbouras13 created, and was due to a very old version of Acanthophis being installed from pip, if I remember correctly.

[kdm9]

Using environment-setup.yml I could not run test as well.

SSH to container,

docker run -it --rm condaforge/miniforge3

Then run

git clone https://github.com/kdm9/Acanthophis.git

(acanthophis-tests) root@5110a2faf4b9:/Acanthophis/tests# mamba env update -f environment-setup.yml
conda-forge/linux-aarch64                                   Using cache
conda-forge/noarch                                          Using cache
bioconda/linux-aarch64                                      Using cache
bioconda/noarch                                             Using cache


Looking for: ["python[version='>=3.11']", "snakemake[version='>=8.2']", "snakemake-interface-executor-plugins[version='>=8.2']", 'tree', 'pip', 'numpy', 'wgsim', 'centrifuge-core', 'samtools', 'msprime', 'curl']


Could not solve for environment specs
The following packages are incompatible
├─ centrifuge-core does not exist (perhaps a typo or a missing channel);
├─ msprime does not exist (perhaps a typo or a missing channel);
├─ samtools does not exist (perhaps a typo or a missing channel);
└─ wgsim does not exist (perhaps a typo or a missing channel).

I tried other methods to install msprime but it does not work,

(acanthophis-tests) root@5110a2faf4b9:/Acanthophis/tests# conda install -c conda-forge msprime
Channels:
 - conda-forge
 - bioconda
Platform: linux-aarch64
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - msprime

I just tried to reproduce this using exactly what you ran, but it worked fine for me with that docker container. Can you confirm that you have pulled the latest condaforge/miniforge3 (see below)? Are there any network or other issues? can you install some conda packages you know to work?

I have the following docker image tag:

$ docker image list
REPOSITORY                     TAG       IMAGE ID       CREATED        SIZE
condaforge/miniforge3          latest    9f2a13a1d04f   5 days ago     395MB

[bricoletc]

@kdm9 , I've tried running the steps here: https://github.com/kdm9/Acanthophis/tree/main/tests
At the step snakemake --snakefile Snakefile.generate-rawdata -j 10 --software-deployment-method conda when running bash setup.sh, I get the following error:

Error in rule ncbitax:
    jobid: 76
    output: rawdata/ncbitax
    log: rawdata/ncbitax/get.log (check log file(s) for error details)
    shell:
        (mkdir -p rawdata/ncbitax && cd rawdata/ncbitax && wget -c ftp://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz && tar -zxvf taxdump.tar.gz) &>rawdata/ncbitax/get.log
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Here is the corresponding log file from that rule:
get.log

Also, I've proposed pointing to the tests in your README in #20 :))

[kdm9]

@bricoletc OK, that's super strange, the log file you attach seems to suggest a failed download (see how it continues past 100% of file downloaded in the progress indicator), resulting in an invalid tar file. I just tried that URL download manually myself and it worked fine.

Any chance you could try wget ftp://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz yourself? I struggle to see how anything Acanthophis does could change what file gets downloaded, so perhaps there was an intermittent error/update in progress on NCBI's end. Out of an abundance of caution, I've changed the old ftp:// link for https://, in case there is there is some odd interaction with FTP. You could re-try with the latest commit.

Many thanks for the readme update b.t.w.! :)

best,
K

[bricoletc]

@kdm9 welcome for the update!
Can confirm it must've been a temporary issue on the NCBI side, now works.

However I still get the same error, on acanthophis version 0.3.3.dev0 installed from pypi as per test.sh:

❯ snakemake -j $(nproc 2>/dev/null || echo 2)  --software-deployment-method conda --ri
Traceback (most recent call last):
  File "/home/adminbrice/Softs/miniforge3/envs/acanthophis/lib/python3.12/site-packages/snakemake/cli.py", line 1893, in args_to_api
    dag_api = workflow_api.dag(
              ^^^^^^^^^^^^^^^^^
  File "/home/adminbrice/Softs/miniforge3/envs/acanthophis/lib/python3.12/site-packages/snakemake/api.py", line 326, in dag
    return DAGApi(
           ^^^^^^^
  File "<string>", line 6, in __init__
  File "/home/adminbrice/Softs/miniforge3/envs/acanthophis/lib/python3.12/site-packages/snakemake/api.py", line 436, in __post_init__
    self.workflow_api._workflow.dag_settings = self.dag_settings
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adminbrice/Softs/miniforge3/envs/acanthophis/lib/python3.12/site-packages/snakemake/api.py", line 383, in _workflow
    workflow.include(
  File "/home/adminbrice/Softs/miniforge3/envs/acanthophis/lib/python3.12/site-packages/snakemake/workflow.py", line 1379, in include
    exec(compile(code, snakefile.get_path_or_uri(), "exec"), self.globals)
  File "/home/adminbrice/Desktop/research/writing/reviewing_exernal/JOSS/Acanthophis/test_tool/Acanthophis/tests/Snakefile", line 6, in <module>
AttributeError: module 'acanthophis' has no attribute 'populate_metadata'

[kdm9]

@bricoletc thanks once again for helping debug this, but the only cause I can think of is an outdated copy of the snakefile in ./tests/. As you can see in the source, the Snakefile directly calls populate_metadata(), which is defined in rules/base.rules.

So, it's certainly true that module 'acanthophis' has no attribute 'populate_metadata', but nor does it need to, since at least 2022. But, if I remember correctly, when you first installed Acanthophis the last released (i.e. non-release candidate) version was from before this change. acanthophis-init by default does not overwrite files, so if you generated that snakefile way back when you first installed Acanthophis, you might be stuck with an old version.

This is suboptimal, so I've just added code to tests/test.sh to nuke all acanthophis code before running acanthophis-init, so that it generates them afresh. This then matches what happens both in CI and in my local precommit hook tests, which happen in ephemeral docker containers.

So, can I please ask one last debugging check of you? Please check if the Snakefile in the tests directory matches HEAD (e.g., from the root of the cloned repo: diff tests/Snakefile acanthophis/template/workflow/Snakefile). If there is a difference, please pull the latest commit in main with the updated test.sh, and then re-run.

Thanks a million for your persistence!

best,
K

[bricoletc]

@kdm9, you were right! I checked out tip of main, and running test.sh now completed successfully! 🚀

[kdm9]

Brilliant, many thanks @bricoletc!!