Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bgcflow run -n command #358

Open
coderbins opened this issue Sep 23, 2024 · 11 comments
Open

bgcflow run -n command #358

coderbins opened this issue Sep 23, 2024 · 11 comments

Comments

@coderbins
Copy link

coderbins commented Sep 23, 2024

(bgcflow) [n@login01 bgcflow]$ bgcflow init -n
Usage: bgcflow init [OPTIONS]
Try 'bgcflow init -h' for help.

Error: No such option: -n
(bgcflow) [n@login01 bgcflow]$ bgcflow init
DEBUG	23/09 12:30:14   Found config file at: /home/n/bgcflow/bgcflow/config/config.yaml
DEBUG	23/09 12:30:14   Creating Project from file /home/n/bgcflow/bgcflow/config/Lactobacillus_delbrueckii/project_config.yaml
DEBUG	23/09 12:30:14   Creating a Project from a YAML file: /home/n/bgcflow/bgcflow/config/Lactobacillus_delbrueckii/project_config.yaml
DEBUG	23/09 12:30:14   Raw (/home/n/bgcflow/bgcflow/config/Lactobacillus_delbrueckii/project_config.yaml) config data: {'name': 'Lactobacillus_delbrueckii', 'pep_version': '2.1.0', 'description': 'Lactobacillus delbrueckii 27 01 2023', 'sample_table': 'samples.csv', 'rules': {'seqfu': True, 'mash': True, 'fastani': True, 'checkm': True, 'gtdbtk': False, 'prokka-gbk': True, 'antismash': True, 'query-bigslice': False, 'bigscape': True, 'bigslice': True, 'automlst-wrapper': True, 'arts': True, 'roary': False, 'eggnog': False, 'eggnog-roary': False, 'deeptfactor': True, 'deeptfactor-roary': False, 'cblaster-genome': True, 'cblaster-bgc': False, 'gecco': True, 'amrfinderplus': False}}
DEBUG	23/09 12:30:14   Adding attributes: name, pep_version, description, sample_table, rules
DEBUG	23/09 12:30:14   Ensuring absolute path for 'samples.csv'
DEBUG	23/09 12:30:14   config_dirpath: /home/n/bgcflow/bgcflow/config/Lactobacillus_delbrueckii
DEBUG	23/09 12:30:14   Expanded and/or made absolute: /home/n/bgcflow/bgcflow/config/Lactobacillus_delbrueckii/samples.csv
DEBUG	23/09 12:30:14   Setting 'sample_table' to '/home/n/bgcflow/bgcflow/config/Lactobacillus_delbrueckii/samples.csv'
DEBUG	23/09 12:30:14   No 'subsample_table' section in configuration file: /home/n/bgcflow/bgcflow/config/Lactobacillus_delbrueckii/project_config.yaml
DEBUG	23/09 12:30:14   No subsample_table specified
DEBUG	23/09 12:30:14   Sample data: OrderedDict([('genome_id', 'GCA_000056065.1'), ('source', 'ncbi'), ('organism', None), ('genus', None), ('species', None), ('strain', None), ('closest_placement_reference', None), ('input_file', None)])
DEBUG	23/09 12:30:14   Sample data: OrderedDict([('genome_id', 'GCA_000182835.1'), ('source', 'ncbi'), ('organism', None), ('genus', None), ('species', None), ('strain', None), ('closest_placement_reference', None), ('input_file', None)])
DEBUG	23/09 12:30:14   Sample data: OrderedDict([('genome_id', 'GCA_000191165.1'), ('source', 'ncbi'), ('organism', None), ('genus', None), ('species', None), ('strain', None), ('closest_placement_reference', None), ('input_file', None)])
DEBUG	23/09 12:30:14   Sample data: OrderedDict([('genome_id', 'GCA_000014405.1'), ('source', 'ncbi'), ('organism', None), ('genus', None), ('species', None), ('strain', None), ('closest_placement_reference', None), ('input_file', None)])
DEBUG	23/09 12:30:14   Checking existence: None
DEBUG	23/09 12:30:14   Checking existence: remove
DEBUG	23/09 12:30:14   Checking existence: append
DEBUG	23/09 12:30:14   Checking existence: duplicate
DEBUG	23/09 12:30:14   Checking existence: imply
DEBUG	23/09 12:30:14   No subsample_table found, skipping merge
DEBUG	23/09 12:30:14   Checking existence: derive
DEBUG	23/09 12:30:14   Checking existence: None
DEBUG	23/09 12:30:14   Setting sample_table index to: ['genome_id']
Available projects:
 - Lactobacillus_delbrueckii : /home/n/bgcflow/bgcflow/config/Lactobacillus_delbrueckii/samples.csv

Do a test run by: `bgcflow run -n`
(bgcflow) [n@login01 bgcflow]$ bgcflow run -n
Monitoring BGCFlow jobs with Panoptes...
DEBUG	23/09 12:30:32   Starting new HTTP connection (1): 127.0.0.1:5000
DEBUG	23/09 12:30:32   http://127.0.0.1:5000 "GET /api/service-info HTTP/11" 200 21
Panoptes already running on http://127.0.0.1:5000
Connecting to Panoptes...
DEBUG	23/09 12:30:32   Starting new HTTP connection (1): 127.0.0.1:5000
DEBUG	23/09 12:30:32   http://127.0.0.1:5000 "GET /api/service-info HTTP/11" 200 21
Panoptes status: running

DEBUG: Using 8 out of 80 available cores

Running Snakemake with command:
cd . && snakemake --snakefile workflow/Snakefile --use-conda --keep-going --rerun-incomplete --rerun-triggers mtime -c 8 --dryrun 	--wms-monitor http://127.0.0.1:5000
Traceback (most recent call last):

  File "/home/n/anaconda3/envs/bgcflow/lib/python3.11/site-packages/requests/models.py", line 974, in json
	return complexjson.loads(self.text, **kwargs)
       	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/n/anaconda3/envs/bgcflow/lib/python3.11/json/__init__.py", line 346, in loads
	return _default_decoder.decode(s)
       	^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/n/anaconda3/envs/bgcflow/lib/python3.11/json/decoder.py", line 337, in decode
	obj, end = self.raw_decode(s, idx=_w(s, 0).end())
           	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/n/anaconda3/envs/bgcflow/lib/python3.11/json/decoder.py", line 355, in raw_decode
	raise JSONDecodeError("Expecting value", s, err.value) from None

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "/home/n/anaconda3/envs/bgcflow/lib/python3.11/site-packages/snakemake/cli.py", line 2155, in main
	success = args_to_api(args, parser)
          	^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/n/anaconda3/envs/bgcflow/lib/python3.11/site-packages/snakemake/cli.py", line 1896, in args_to_api
	log_handlers = setup_log_handlers(args, parser)
               	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/n/anaconda3/envs/bgcflow/lib/python3.11/site-packages/snakemake/cli.py", line 1807, in setup_log_handlers
	wms_logger = logging.WMSLogger(
             	^^^^^^^^^^^^^^^^^^

  File "/home/n/anaconda3/envs/bgcflow/lib/python3.11/site-packages/snakemake/logging.py", line 148, in __init__
	self.create_workflow()

  File "/home/n/anaconda3/envs/bgcflow/lib/python3.11/site-packages/snakemake/logging.py", line 199, in create_workflow
	id = response.json()["id"]
     	^^^^^^^^^^^^^^^

  File "/home/n/anaconda3/envs/bgcflow/lib/python3.11/site-packages/requests/models.py", line 978, in json
	raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)

requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
@matinnuhamunada
Copy link
Collaborator

Hi @coderbins, from the error message, I can see that you have successfully created a project configuration from the template but failed to run the project.

I cannot reproduce this error, but it's either an error reading the project config or problem with panoptes (the monitoring workflow).

What I can think about is:

  1. Did you make any changes to the configuration files before running the workflow? If yes, can you share it here so we can debug it together?
  2. In case there is an error in running panoptes, try running it using the vanilla snakemake command and see if it works?:
snakemake --snakefile workflow/Snakefile --use-conda --keep-going --rerun-incomplete --rerun-triggers mtime -c 8 --dryrun 
  1. Can you give me your system detail? Which linux version etc. And maybe export the conda environment and share it here (using conda env export > coderbins.yaml)

@coderbins
Copy link
Author

coderbins commented Sep 23, 2024 via email

@matinnuhamunada
Copy link
Collaborator

matinnuhamunada commented Sep 23, 2024

Have you tried following the instruction in the error message?

The 'mamba' command is not available in the shell /usr/bin/bash that will
be used by Snakemake. You have to ensure that it is in your PATH, e.g.,
first activating the conda base environment with conda activate base.The
mamba package manager (https://github.com/mamba-org/mamba) is a fast and
robust conda replacement. It is the recommended way of using Snakemake's
conda integration. It can be installed with conda install -n base -c conda-forge mamba. If you still prefer to use conda, you can enforce that
by setting --conda-frontend conda.

As described in the installation wiki: https://github.com/NBChub/bgcflow/wiki/00-Installation-Guide#conda-package-manager, we recommend using https://github.com/conda-forge/miniforge#mambaforge rather than anaconda or conda installer

@coderbins
Copy link
Author

coderbins commented Sep 24, 2024 via email

@matinnuhamunada
Copy link
Collaborator

matinnuhamunada commented Sep 24, 2024

Hi, this is likely an issue from Anaconda. Please refer to the mamba installation here: https://mamba.readthedocs.io/en/latest/installation/mamba-installation.html

I would personally uninstall Anaconda and replace it with miniforge as suggested in the Wiki.

wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge-pypy3-Linux-x86_64.sh"
bash Miniforge-pypy3-Linux-x86_64.sh

@coderbins
Copy link
Author

coderbins commented Sep 25, 2024

Thank you so much. Its working with mamba. but when i try to use 'bgcflow build report'. i am facing certain errors again.

Activating conda environment: .snakemake/conda/6108734d128c0fa773849e795b21d0f5_
[Wed Sep 25 15:14:37 2024]
Error in rule mkdocs_py_report:
jobid: 23
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/deeptfactor.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/deeptfactor.md
log: logs/report/deeptfactor-report-Lactobacillus_delbrueckii.log (check log file(s) for error details)
conda-env: /home/simrann/bgcflow/.snakemake/conda/6108734d128c0fa773849e795b21d0f5_
shell:

        jupyter nbconvert --to markdown --execute data/processed/Lactobacillus_delbrueckii/docs/deeptfactor.ipynb --no-input --output deeptfactor.md 2>> logs/report/deeptfactor-report-Lactobacillus_delbrueckii.log
        
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Select jobs to execute...
Execute 1 jobs...

[Wed Sep 25 15:14:37 2024]
localrule mkdocs_py_report:
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/checkm.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/checkm.md
log: logs/report/checkm-report-Lactobacillus_delbrueckii.log
jobid: 9
reason: Missing output files: data/processed/Lactobacillus_delbrueckii/docs/checkm.md; Input files updated by another job: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/checkm.ipynb
wildcards: name=Lactobacillus_delbrueckii, bgcflow_rules_py=checkm
resources: tmpdir=/tmp

Activating conda environment: .snakemake/conda/6108734d128c0fa773849e795b21d0f5_
[Wed Sep 25 15:14:41 2024]
Error in rule mkdocs_py_report:
jobid: 15
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/bigscape.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/bigscape.md
log: logs/report/bigscape-report-Lactobacillus_delbrueckii.log (check log file(s) for error details)
conda-env: /home/simrann/bgcflow/.snakemake/conda/6108734d128c0fa773849e795b21d0f5_
shell:

        jupyter nbconvert --to markdown --execute data/processed/Lactobacillus_delbrueckii/docs/bigscape.ipynb --no-input --output bigscape.md 2>> logs/report/bigscape-report-Lactobacillus_delbrueckii.log
        
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Select jobs to execute...
Execute 1 jobs...

[Wed Sep 25 15:14:41 2024]
localrule mkdocs_py_report:
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/prokka-gbk.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/prokka-gbk.md
log: logs/report/prokka-gbk-report-Lactobacillus_delbrueckii.log
jobid: 11
reason: Missing output files: data/processed/Lactobacillus_delbrueckii/docs/prokka-gbk.md; Input files updated by another job: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/prokka-gbk.ipynb
wildcards: name=Lactobacillus_delbrueckii, bgcflow_rules_py=prokka-gbk
resources: tmpdir=/tmp

Activating conda environment: .snakemake/conda/6108734d128c0fa773849e795b21d0f5_
[Wed Sep 25 15:14:41 2024]
Error in rule mkdocs_py_report:
jobid: 27
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/gecco.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/gecco.md
log: logs/report/gecco-report-Lactobacillus_delbrueckii.log (check log file(s) for error details)
conda-env: /home/simrann/bgcflow/.snakemake/conda/6108734d128c0fa773849e795b21d0f5_
shell:

        jupyter nbconvert --to markdown --execute data/processed/Lactobacillus_delbrueckii/docs/gecco.ipynb --no-input --output gecco.md 2>> logs/report/gecco-report-Lactobacillus_delbrueckii.log
        
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Select jobs to execute...
Execute 1 jobs...

[Wed Sep 25 15:14:41 2024]
localrule mkdocs_py_report:
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/seqfu.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/seqfu.md
log: logs/report/seqfu-report-Lactobacillus_delbrueckii.log
jobid: 1
reason: Missing output files: data/processed/Lactobacillus_delbrueckii/docs/seqfu.md; Input files updated by another job: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/seqfu.ipynb
wildcards: name=Lactobacillus_delbrueckii, bgcflow_rules_py=seqfu
resources: tmpdir=/tmp

Activating conda environment: .snakemake/conda/6108734d128c0fa773849e795b21d0f5_
[Wed Sep 25 15:14:41 2024]
Error in rule mkdocs_py_report:
jobid: 13
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/antismash.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/antismash.md
log: logs/report/antismash-report-Lactobacillus_delbrueckii.log (check log file(s) for error details)
conda-env: /home/simrann/bgcflow/.snakemake/conda/6108734d128c0fa773849e795b21d0f5_
shell:

        jupyter nbconvert --to markdown --execute data/processed/Lactobacillus_delbrueckii/docs/antismash.ipynb --no-input --output antismash.md 2>> logs/report/antismash-report-Lactobacillus_delbrueckii.log
        
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Select jobs to execute...
Execute 1 jobs...

[Wed Sep 25 15:14:41 2024]
localrule mkdocs_py_report:
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/bigslice.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/bigslice.md
log: logs/report/bigslice-report-Lactobacillus_delbrueckii.log
jobid: 17
reason: Missing output files: data/processed/Lactobacillus_delbrueckii/docs/bigslice.md; Input files updated by another job: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/bigslice.ipynb
wildcards: name=Lactobacillus_delbrueckii, bgcflow_rules_py=bigslice
resources: tmpdir=/tmp

Activating conda environment: .snakemake/conda/6108734d128c0fa773849e795b21d0f5_
[Wed Sep 25 15:14:42 2024]
Error in rule mkdocs_py_report:
jobid: 21
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/arts.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/arts.md
log: logs/report/arts-report-Lactobacillus_delbrueckii.log (check log file(s) for error details)
conda-env: /home/simrann/bgcflow/.snakemake/conda/6108734d128c0fa773849e795b21d0f5_
shell:

        jupyter nbconvert --to markdown --execute data/processed/Lactobacillus_delbrueckii/docs/arts.ipynb --no-input --output arts.md 2>> logs/report/arts-report-Lactobacillus_delbrueckii.log
        
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Select jobs to execute...
Execute 1 jobs...

[Wed Sep 25 15:14:42 2024]
localrule mkdocs_py_report:
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/cblaster-genome.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/cblaster-genome.md
log: logs/report/cblaster-genome-report-Lactobacillus_delbrueckii.log
jobid: 25
reason: Missing output files: data/processed/Lactobacillus_delbrueckii/docs/cblaster-genome.md; Input files updated by another job: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/cblaster-genome.ipynb
wildcards: name=Lactobacillus_delbrueckii, bgcflow_rules_py=cblaster-genome
resources: tmpdir=/tmp

Activating conda environment: .snakemake/conda/6108734d128c0fa773849e795b21d0f5_
[Wed Sep 25 15:14:42 2024]
Error in rule mkdocs_py_report:
jobid: 5
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/mash.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/mash.md
log: logs/report/mash-report-Lactobacillus_delbrueckii.log (check log file(s) for error details)
conda-env: /home/simrann/bgcflow/.snakemake/conda/6108734d128c0fa773849e795b21d0f5_
shell:

        jupyter nbconvert --to markdown --execute data/processed/Lactobacillus_delbrueckii/docs/mash.ipynb --no-input --output mash.md 2>> logs/report/mash-report-Lactobacillus_delbrueckii.log
        
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Wed Sep 25 15:14:42 2024]
Error in rule mkdocs_py_report:
jobid: 7
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/fastani.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/fastani.md
log: logs/report/fastani-report-Lactobacillus_delbrueckii.log (check log file(s) for error details)
conda-env: /home/simrann/bgcflow/.snakemake/conda/6108734d128c0fa773849e795b21d0f5_
shell:

        jupyter nbconvert --to markdown --execute data/processed/Lactobacillus_delbrueckii/docs/fastani.ipynb --no-input --output fastani.md 2>> logs/report/fastani-report-Lactobacillus_delbrueckii.log
        
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Wed Sep 25 15:14:45 2024]
Error in rule mkdocs_rpy_report:
jobid: 19
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/automlst-wrapper.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/automlst-wrapper.md
log: logs/report/automlst-wrapper-report-Lactobacillus_delbrueckii.log (check log file(s) for error details)
conda-env: /home/simrann/bgcflow/.snakemake/conda/51620c99e7ffd92732a8d9f8a0d02483_
shell:

        jupyter nbconvert --to markdown --execute data/processed/Lactobacillus_delbrueckii/docs/automlst-wrapper.ipynb --no-input --output automlst-wrapper.md 2>> logs/report/automlst-wrapper-report-Lactobacillus_delbrueckii.log
        
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Wed Sep 25 15:14:49 2024]
Error in rule mkdocs_py_report:
jobid: 9
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/checkm.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/checkm.md
log: logs/report/checkm-report-Lactobacillus_delbrueckii.log (check log file(s) for error details)
conda-env: /home/simrann/bgcflow/.snakemake/conda/6108734d128c0fa773849e795b21d0f5_
shell:

        jupyter nbconvert --to markdown --execute data/processed/Lactobacillus_delbrueckii/docs/checkm.ipynb --no-input --output checkm.md 2>> logs/report/checkm-report-Lactobacillus_delbrueckii.loActivating conda environment: .snakemake/conda/6108734d128c0fa773849e795b21d0f5_

[Wed Sep 25 15:14:37 2024]
Error in rule mkdocs_py_report:
jobid: 23
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/deeptfactor.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/deeptfactor.md
log: logs/report/deeptfactor-report-Lactobacillus_delbrueckii.log (check log file(s) for error details)
conda-env: /home/simrann/bgcflow/.snakemake/conda/6108734d128c0fa773849e795b21d0f5_
shell:

        jupyter nbconvert --to markdown --execute data/processed/Lactobacillus_delbrueckii/docs/deeptfactor.ipynb --no-input --output deeptfactor.md 2>> logs/report/deeptfactor-report-Lactobacillus_delbrueckii.log
        
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Select jobs to execute...
Execute 1 jobs...

[Wed Sep 25 15:14:37 2024]
localrule mkdocs_py_report:
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/checkm.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/checkm.md
log: logs/report/checkm-report-Lactobacillus_delbrueckii.log
jobid: 9
reason: Missing output files: data/processed/Lactobacillus_delbrueckii/docs/checkm.md; Input files updated by another job: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/checkm.ipynb
wildcards: name=Lactobacillus_delbrueckii, bgcflow_rules_py=checkm
resources: tmpdir=/tmp

Activating conda environment: .snakemake/conda/6108734d128c0fa773849e795b21d0f5_
[Wed Sep 25 15:14:41 2024]
Error in rule mkdocs_py_report:
jobid: 15
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/bigscape.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/bigscape.md
log: logs/report/bigscape-report-Lactobacillus_delbrueckii.log (check log file(s) for error details)
conda-env: /home/simrann/bgcflow/.snakemake/conda/6108734d128c0fa773849e795b21d0f5_
shell:

        jupyter nbconvert --to markdown --execute data/processed/Lactobacillus_delbrueckii/docs/bigscape.ipynb --no-input --output bigscape.md 2>> logs/report/bigscape-report-Lactobacillus_delbrueckii.log
        
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Select jobs to execute...
Execute 1 jobs...

[Wed Sep 25 15:14:41 2024]
localrule mkdocs_py_report:
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/prokka-gbk.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/prokka-gbk.md
log: logs/report/prokka-gbk-report-Lactobacillus_delbrueckii.log
jobid: 11
reason: Missing output files: data/processed/Lactobacillus_delbrueckii/docs/prokka-gbk.md; Input files updated by another job: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/prokka-gbk.ipynb
wildcards: name=Lactobacillus_delbrueckii, bgcflow_rules_py=prokka-gbk
resources: tmpdir=/tmp

Activating conda environment: .snakemake/conda/6108734d128c0fa773849e795b21d0f5_
[Wed Sep 25 15:14:41 2024]
Error in rule mkdocs_py_report:
jobid: 27
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/gecco.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/gecco.md
log: logs/report/gecco-report-Lactobacillus_delbrueckii.log (check log file(s) for error details)
conda-env: /home/simrann/bgcflow/.snakemake/conda/6108734d128c0fa773849e795b21d0f5_
shell:

        jupyter nbconvert --to markdown --execute data/processed/Lactobacillus_delbrueckii/docs/gecco.ipynb --no-input --output gecco.md 2>> logs/report/gecco-report-Lactobacillus_delbrueckii.log
        
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Select jobs to execute...
Execute 1 jobs...

[Wed Sep 25 15:14:41 2024]
localrule mkdocs_py_report:
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/seqfu.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/seqfu.md
log: logs/report/seqfu-report-Lactobacillus_delbrueckii.log
jobid: 1
reason: Missing output files: data/processed/Lactobacillus_delbrueckii/docs/seqfu.md; Input files updated by another job: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/seqfu.ipynb
wildcards: name=Lactobacillus_delbrueckii, bgcflow_rules_py=seqfu
resources: tmpdir=/tmp

Activating conda environment: .snakemake/conda/6108734d128c0fa773849e795b21d0f5_
[Wed Sep 25 15:14:41 2024]
Error in rule mkdocs_py_report:
jobid: 13
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/antismash.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/antismash.md
log: logs/report/antismash-report-Lactobacillus_delbrueckii.log (check log file(s) for error details)
conda-env: /home/simrann/bgcflow/.snakemake/conda/6108734d128c0fa773849e795b21d0f5_
shell:

        jupyter nbconvert --to markdown --execute data/processed/Lactobacillus_delbrueckii/docs/antismash.ipynb --no-input --output antismash.md 2>> logs/report/antismash-report-Lactobacillus_delbrueckii.log
        
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Select jobs to execute...
Execute 1 jobs...

[Wed Sep 25 15:14:41 2024]
localrule mkdocs_py_report:
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/bigslice.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/bigslice.md
log: logs/report/bigslice-report-Lactobacillus_delbrueckii.log
jobid: 17
reason: Missing output files: data/processed/Lactobacillus_delbrueckii/docs/bigslice.md; Input files updated by another job: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/bigslice.ipynb
wildcards: name=Lactobacillus_delbrueckii, bgcflow_rules_py=bigslice
resources: tmpdir=/tmp

Activating conda environment: .snakemake/conda/6108734d128c0fa773849e795b21d0f5_
[Wed Sep 25 15:14:42 2024]
Error in rule mkdocs_py_report:
jobid: 21
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/arts.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/arts.md
log: logs/report/arts-report-Lactobacillus_delbrueckii.log (check log file(s) for error details)
conda-env: /home/simrann/bgcflow/.snakemake/conda/6108734d128c0fa773849e795b21d0f5_
shell:

        jupyter nbconvert --to markdown --execute data/processed/Lactobacillus_delbrueckii/docs/arts.ipynb --no-input --output arts.md 2>> logs/report/arts-report-Lactobacillus_delbrueckii.log
        
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Select jobs to execute...
Execute 1 jobs...

[Wed Sep 25 15:14:42 2024]
localrule mkdocs_py_report:
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/cblaster-genome.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/cblaster-genome.md
log: logs/report/cblaster-genome-report-Lactobacillus_delbrueckii.log
jobid: 25
reason: Missing output files: data/processed/Lactobacillus_delbrueckii/docs/cblaster-genome.md; Input files updated by another job: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/cblaster-genome.ipynb
wildcards: name=Lactobacillus_delbrueckii, bgcflow_rules_py=cblaster-genome
resources: tmpdir=/tmp

Activating conda environment: .snakemake/conda/6108734d128c0fa773849e795b21d0f5_
[Wed Sep 25 15:14:42 2024]
Error in rule mkdocs_py_report:
jobid: 5
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/mash.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/mash.md
log: logs/report/mash-report-Lactobacillus_delbrueckii.log (check log file(s) for error details)
conda-env: /home/simrann/bgcflow/.snakemake/conda/6108734d128c0fa773849e795b21d0f5_
shell:

        jupyter nbconvert --to markdown --execute data/processed/Lactobacillus_delbrueckii/docs/mash.ipynb --no-input --output mash.md 2>> logs/report/mash-report-Lactobacillus_delbrueckii.log
        
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Wed Sep 25 15:14:42 2024]
Error in rule mkdocs_py_report:
jobid: 7
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/fastani.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/fastani.md
log: logs/report/fastani-report-Lactobacillus_delbrueckii.log (check log file(s) for error details)
conda-env: /home/simrann/bgcflow/.snakemake/conda/6108734d128c0fa773849e795b21d0f5_
shell:

        jupyter nbconvert --to markdown --execute data/processed/Lactobacillus_delbrueckii/docs/fastani.ipynb --no-input --output fastani.md 2>> logs/report/fastani-report-Lactobacillus_delbrueckii.log
        
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Wed Sep 25 15:14:45 2024]
Error in rule mkdocs_rpy_report:
jobid: 19
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/automlst-wrapper.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/automlst-wrapper.md
log: logs/report/automlst-wrapper-report-Lactobacillus_delbrueckii.log (check log file(s) for error details)
conda-env: /home/simrann/bgcflow/.snakemake/conda/51620c99e7ffd92732a8d9f8a0d02483_
shell:

        jupyter nbconvert --to markdown --execute data/processed/Lactobacillus_delbrueckii/docs/automlst-wrapper.ipynb --no-input --output automlst-wrapper.md 2>> logs/report/automlst-wrapper-report-Lactobacillus_delbrueckii.log
        
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Wed Sep 25 15:14:49 2024]
Error in rule mkdocs_py_report:
jobid: 9
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/checkm.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/checkm.md
log: logs/report/checkm-report-Lactobacillus_delbrueckii.log (check log file(s) for error details)
conda-env: /home/simrann/bgcflow/.snakemake/conda/6108734d128c0fa773849e795b21d0f5_
shell:

        jupyter nbconvert --to markdown --execute data/processed/Lactobacillus_delbrueckii/docs/checkm.ipynb --no-input --output checkm.md 2>> logs/report/checkm-report-Lactobacillus_delbrueckii.log
        
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Wed Sep 25 15:14:49 2024]
Finished job 17.
17 of 30 steps (57%) done
[Wed Sep 25 15:14:50 2024]
Finished job 25.
18 of 30 steps (60%) done
[Wed Sep 25 15:14:50 2024]
Error in rule mkdocs_py_report:
jobid: 1
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/seqfu.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/seqfu.md
log: logs/report/seqfu-report-Lactobacillus_delbrueckii.log (check log file(s) for error details)
conda-env: /home/simrann/bgcflow/.snakemake/conda/6108734d128c0fa773849e795b21d0f5_
shell:

        jupyter nbconvert --to markdown --execute data/processed/Lactobacillus_delbrueckii/docs/seqfu.ipynb --no-input --output seqfu.md 2>> logs/report/seqfu-report-Lactobacillus_delbrueckii.log
        
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Wed Sep 25 15:14:50 2024]
Error in rule mkdocs_py_report:
jobid: 11
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/prokka-gbk.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/prokka-gbk.md
log: logs/report/prokka-gbk-report-Lactobacillus_delbrueckii.log (check log file(s) for error details)
conda-env: /home/simrann/bgcflow/.snakemake/conda/6108734d128c0fa773849e795b21d0f5_
shell:

        jupyter nbconvert --to markdown --execute data/processed/Lactobacillus_delbrueckii/docs/prokka-gbk.ipynb --no-input --output prokka-gbk.md 2>> logs/report/prokka-gbk-report-Lactobacillus_delbrueckii.log
        
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-09-25T145500.383306.snakemake.log
WorkflowError:
At least one job did not complete successfully.
g

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Wed Sep 25 15:14:49 2024]
Finished job 17.
17 of 30 steps (57%) done
[Wed Sep 25 15:14:50 2024]
Finished job 25.
18 of 30 steps (60%) done
[Wed Sep 25 15:14:50 2024]
Error in rule mkdocs_py_report:
jobid: 1
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/seqfu.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/seqfu.md
log: logs/report/seqfu-report-Lactobacillus_delbrueckii.log (check log file(s) for error details)
conda-env: /home/simrann/bgcflow/.snakemake/conda/6108734d128c0fa773849e795b21d0f5_
shell:

        jupyter nbconvert --to markdown --execute data/processed/Lactobacillus_delbrueckii/docs/seqfu.ipynb --no-input --output seqfu.md 2>> logs/report/seqfu-report-Lactobacillus_delbrueckii.log
        
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Wed Sep 25 15:14:50 2024]
Error in rule mkdocs_py_report:
jobid: 11
input: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json, data/processed/Lactobacillus_delbrueckii/docs/prokka-gbk.ipynb
output: data/processed/Lactobacillus_delbrueckii/docs/prokka-gbk.md
log: logs/report/prokka-gbk-report-Lactobacillus_delbrueckii.log (check log file(s) for error details)
conda-env: /home/simrann/bgcflow/.snakemake/conda/6108734d128c0fa773849e795b21d0f5_
shell:

        jupyter nbconvert --to markdown --execute data/processed/Lactobacillus_delbrueckii/docs/prokka-gbk.ipynb --no-input --output prokka-gbk.md 2>> logs/report/prokka-gbk-report-Lactobacillus_delbrueckii.log
        
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-09-25T145500.383306.snakemake.log
WorkflowError:
At least one job did not complete successfully.

@matinnuhamunada
Copy link
Collaborator

Glad that it works!

To make the report, you need to finish the analysis first. Which means you need to run bgcflow run without the -n (--dry-run).
Is that the case?

@coderbins
Copy link
Author

i did try on mamba but there were issues. is their a docker containerfor bgcflow?
now it is showing this: (bgcflow) [simrann@login01 bgcflow]$ bgcflow run

Monitoring BGCFlow jobs with Panoptes...
DEBUG 28/09 15:15:47 Starting new HTTP connection (1): 127.0.0.1:5000
DEBUG 28/09 15:15:47 http://127.0.0.1:5000 "GET /api/service-info HTTP/11" 200 21
Panoptes already running on http://127.0.0.1:5000
Connecting to Panoptes...
DEBUG 28/09 15:15:47 Starting new HTTP connection (1): 127.0.0.1:5000
DEBUG 28/09 15:15:47 http://127.0.0.1:5000 "GET /api/service-info HTTP/11" 200 21
Panoptes status: running

DEBUG: Using 8 out of 80 available cores

Running Snakemake with command:
cd . && snakemake --snakefile workflow/Snakefile --use-conda --keep-going --rerun-incomplete --rerun-triggers mtime -c 8 --wms-monitor http://127.0.0.1:5000
Traceback (most recent call last):

File "/home/simrann/miniforge-pypy3/envs/bgcflow/lib/python3.11/site-packages/requests/models.py", line 974, in json
return complexjson.loads(self.text, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/home/simrann/miniforge-pypy3/envs/bgcflow/lib/python3.11/json/init.py", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/home/simrann/miniforge-pypy3/envs/bgcflow/lib/python3.11/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/home/simrann/miniforge-pypy3/envs/bgcflow/lib/python3.11/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/home/simrann/miniforge-pypy3/envs/bgcflow/lib/python3.11/site-packages/snakemake/cli.py", line 2155, in main
success = args_to_api(args, parser)
^^^^^^^^^^^^^^^^^^^^^^^^^

File "/home/simrann/miniforge-pypy3/envs/bgcflow/lib/python3.11/site-packages/snakemake/cli.py", line 1896, in args_to_api
log_handlers = setup_log_handlers(args, parser)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/home/simrann/miniforge-pypy3/envs/bgcflow/lib/python3.11/site-packages/snakemake/cli.py", line 1807, in setup_log_handlers
wms_logger = logging.WMSLogger(
^^^^^^^^^^^^^^^^^^

File "/home/simrann/miniforge-pypy3/envs/bgcflow/lib/python3.11/site-packages/snakemake/logging.py", line 148, in init
self.create_workflow()

File "/home/simrann/miniforge-pypy3/envs/bgcflow/lib/python3.11/site-packages/snakemake/logging.py", line 199, in create_workflow
id = response.json()["id"]
^^^^^^^^^^^^^^^

File "/home/simrann/miniforge-pypy3/envs/bgcflow/lib/python3.11/site-packages/requests/models.py", line 978, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)

requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

@coderbins
Copy link
Author

coderbins commented Oct 2, 2024 via email

@matinnuhamunada
Copy link
Collaborator

Hi @coderbins , sorry I was a bit occupied with other stuff.

This seems like a panoptes issue, have you already tried running it without the monitoring workflow?

snakemake --snakefile workflow/Snakefile --use-conda --keep-going
--rerun-incomplete --rerun-triggers mtime -c 8

We are still working on the Docker containerization, but don't expect it to be done before December

@coderbins
Copy link
Author

coderbins commented Oct 11, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants