Possible syntax error regarding a HoverTool attribute and use of iteritems on a Series

[411an13]

Below is a snippet from the log of a recent Qadabra run that seems to have ended due to 2 errors. From what I can tell, the first one occurs in plot_rank_comparison.py in the line hover = HoverTool(mode="mouse", names=["points"], attachment="below") because it was expecting name instead of names. The second seems to occur in plot_pca.py because iteritems is called on a Series instead of items. I checked the log files but they did not shed any more light on the errors.

Given that these might be relatively quick fixes if they are indeed what caused it to error out, could you please let me know if this issue has been encountered before?

Here is the snippet from the log:

Activating conda environment: .snakemake/conda/b93b3e3540dfc9b230066ff51de94cd0_
Activating conda environment: .snakemake/conda/b93b3e3540dfc9b230066ff51de94cd0_
Traceback (most recent call last):
  File "/Users/aphillip/Library/CloudStorage/Desktop/Projects/counts_tables/differential_abundance_analysis/prok_slc_environments/.snakemake/scripts/tmpzhmmn4p4.plot_rank_comparison.py", line 52, in <module>
    hover = HoverTool(mode="mouse", names=["points"], attachment="below")
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aphillip/Library/CloudStorage/Desktop/Projects/counts_tables/differential_abundance_analysis/prok_slc_environments/.snakemake/conda/b93b3e3540dfc9b230066ff51de94cd0_/lib/python3.11/site-packages/bokeh/models/tools.py", line 1279, in __init__
    super().__init__(*args, **kwargs)
  File "/Users/aphillip/Library/CloudStorage/Desktop/Projects/counts_tables/differential_abundance_analysis/prok_slc_environments/.snakemake/conda/b93b3e3540dfc9b230066ff51de94cd0_/lib/python3.11/site-packages/bokeh/models/tools.py", line 346, in __init__
    super().__init__(*args, **kwargs)
  File "/Users/aphillip/Library/CloudStorage/Desktop/Projects/counts_tables/differential_abundance_analysis/prok_slc_environments/.snakemake/conda/b93b3e3540dfc9b230066ff51de94cd0_/lib/python3.11/site-packages/bokeh/models/tools.py", line 256, in __init__
    super().__init__(*args, **kwargs)
  File "/Users/aphillip/Library/CloudStorage/Desktop/Projects/counts_tables/differential_abundance_analysis/prok_slc_environments/.snakemake/conda/b93b3e3540dfc9b230066ff51de94cd0_/lib/python3.11/site-packages/bokeh/models/tools.py", line 181, in __init__
    super().__init__(*args, **kwargs)
  File "/Users/aphillip/Library/CloudStorage/Desktop/Projects/counts_tables/differential_abundance_analysis/prok_slc_environments/.snakemake/conda/b93b3e3540dfc9b230066ff51de94cd0_/lib/python3.11/site-packages/bokeh/model/model.py", line 110, in __init__
    super().__init__(**kwargs)
  File "/Users/aphillip/Library/CloudStorage/Desktop/Projects/counts_tables/differential_abundance_analysis/prok_slc_environments/.snakemake/conda/b93b3e3540dfc9b230066ff51de94cd0_/lib/python3.11/site-packages/bokeh/core/has_props.py", line 302, in __init__
    setattr(self, name, value)
  File "/Users/aphillip/Library/CloudStorage/Desktop/Projects/counts_tables/differential_abundance_analysis/prok_slc_environments/.snakemake/conda/b93b3e3540dfc9b230066ff51de94cd0_/lib/python3.11/site-packages/bokeh/core/has_props.py", line 340, in __setattr__
    self._raise_attribute_error_with_matches(name, properties)
  File "/Users/aphillip/Library/CloudStorage/Desktop/Projects/counts_tables/differential_abundance_analysis/prok_slc_environments/.snakemake/conda/b93b3e3540dfc9b230066ff51de94cd0_/lib/python3.11/site-packages/bokeh/core/has_props.py", line 375, in _raise_attribute_error_with_matches
    raise AttributeError(f"unexpected attribute {name!r} to {self.__class__.__name__}, {text} attributes are {nice_join(matches)}")
AttributeError: unexpected attribute 'names' to HoverTool, similar attributes are name
[Wed Jan 17 16:35:46 2024]
Finished job 19.
115 of 152 steps (76%) done
Select jobs to execute...

[Wed Jan 17 16:35:47 2024]
rule plot_pca:
    input: results/environment_comparisons/pca/pca_features.tsv, results/environment_comparisons/pca/pca_tools.tsv, results/environment_comparisons/pca/proportion_explained.tsv
    output: figures/environment_comparisons/pca.svg
    log: log/environment_comparisons/plot_pca.log
    jobid: 30
    reason: Missing output files: figures/environment_comparisons/pca.svg; Input files updated by another job: results/environment_comparisons/pca/pca_tools.tsv, results/environment_comparisons/pca/proportion_explained.tsv, results/environment_comparisons/pca/pca_features.tsv
    wildcards: dataset=environment_comparisons
    resources: tmpdir=/var/folders/_w/22jzw70d6jx1ngxrkkj2p5g80000gp/T

Activating conda environment: .snakemake/conda/b93b3e3540dfc9b230066ff51de94cd0_
[Wed Jan 17 16:35:47 2024]
Error in rule plot_rank_comparison:
    jobid: 29
    input: results/environment_comparisons/concatenated_differentials.tsv
    output: figures/environment_comparisons/rank_comparisons.html
    log: log/environment_comparisons/plot_rank_comparison.log (check log file(s) for error details)
    conda-env: /Users/aphillip/Library/CloudStorage/Desktop/Projects/counts_tables/differential_abundance_analysis/prok_slc_environments/.snakemake/conda/b93b3e3540dfc9b230066ff51de94cd0_

RuleException:
CalledProcessError in file /Users/aphillip/anaconda3/envs/qadabra_env/lib/python3.10/site-packages/qadabra/workflow/rules/visualization.smk, line 149:
Command 'source /Users/aphillip/anaconda3/bin/activate '/Users/aphillip/Library/CloudStorage/Desktop/Projects/counts_tables/differential_abundance_analysis/prok_slc_environments/.snakemake/conda/b93b3e3540dfc9b230066ff51de94cd0_'; set -euo pipefail;  python /Users/aphillip/Library/CloudStorage/Desktop/Projects/counts_tables/differential_abundance_analysis/prok_slc_environments/.snakemake/scripts/tmpzhmmn4p4.plot_rank_comparison.py' returned non-zero exit status 1.
  File "/Users/aphillip/anaconda3/envs/qadabra_env/lib/python3.10/site-packages/qadabra/workflow/rules/visualization.smk", line 149, in __rule_plot_rank_comparison
  File "/Users/aphillip/anaconda3/envs/qadabra_env/lib/python3.10/concurrent/futures/thread.py", line 58, in run
Activating conda environment: .snakemake/conda/b93b3e3540dfc9b230066ff51de94cd0_
[Wed Jan 17 16:35:48 2024]
Finished job 139.
116 of 152 steps (76%) done
[Wed Jan 17 16:35:50 2024]
Finished job 28.
117 of 152 steps (77%) done
Traceback (most recent call last):
  File "/Users/aphillip/Library/CloudStorage/Desktop/Projects/counts_tables/differential_abundance_analysis/prok_slc_environments/.snakemake/scripts/tmp1th9bjlw.plot_pca.py", line 106, in <module>
    for i, x in prop_exp.iteritems()
                ^^^^^^^^^^^^^^^^^^
  File "/Users/aphillip/Library/CloudStorage/Desktop/Projects/counts_tables/differential_abundance_analysis/prok_slc_environments/.snakemake/conda/b93b3e3540dfc9b230066ff51de94cd0_/lib/python3.11/site-packages/pandas/core/generic.py", line 6204, in __getattr__
    return object.__getattribute__(self, name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Series' object has no attribute 'iteritems'
[Wed Jan 17 16:35:52 2024]
Error in rule plot_pca:
    jobid: 30
    input: results/environment_comparisons/pca/pca_features.tsv, results/environment_comparisons/pca/pca_tools.tsv, results/environment_comparisons/pca/proportion_explained.tsv
    output: figures/environment_comparisons/pca.svg
    log: log/environment_comparisons/plot_pca.log (check log file(s) for error details)
    conda-env: /Users/aphillip/Library/CloudStorage/Desktop/Projects/counts_tables/differential_abundance_analysis/prok_slc_environments/.snakemake/conda/b93b3e3540dfc9b230066ff51de94cd0_

RuleException:
CalledProcessError in file /Users/aphillip/anaconda3/envs/qadabra_env/lib/python3.10/site-packages/qadabra/workflow/rules/visualization.smk, line 131:
Command 'source /Users/aphillip/anaconda3/bin/activate '/Users/aphillip/Library/CloudStorage/Desktop/Projects/counts_tables/differential_abundance_analysis/prok_slc_environments/.snakemake/conda/b93b3e3540dfc9b230066ff51de94cd0_'; set -euo pipefail;  python /Users/aphillip/Library/CloudStorage/Desktop/Projects/counts_tables/differential_abundance_analysis/prok_slc_environments/.snakemake/scripts/tmp1th9bjlw.plot_pca.py' returned non-zero exit status 1.
  File "/Users/aphillip/anaconda3/envs/qadabra_env/lib/python3.10/site-packages/qadabra/workflow/rules/visualization.smk", line 131, in __rule_plot_pca
  File "/Users/aphillip/anaconda3/envs/qadabra_env/lib/python3.10/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-01-17T161851.931421.snakemake.log

[jolespin]

We took a look at this and Bokeh deprecated name for names in version 3.0.0 so this needs to be downgraded to 2.4.3
Also, Pandas deprecated iteritems on pd.Series objects in version 2.0.0 so this needs to be downgraded to 1.5.3

[gibsramen]

Thanks @jolespin & @411an13! @yangchen2 can you try these changes out and see if they address this problem? Relevant to #55

[gibsramen]

@411an13 Was this with the PyPI version of Qadabra? Installing directly from GitHub should, I think, resolve these. I'll try to get an updated version on PyPI soon.

[jolespin]

I'm not sure how snakemake works exactly but are there yaml files for the conda environments used in the backend?

[gibsramen]

We have a folder with several env yamls (https://github.com/biocore/qadabra/tree/main/qadabra/workflow/envs) but most of the dependencies are not pinned.

[jolespin]

We have a folder with several env yamls (https://github.com/biocore/qadabra/tree/main/qadabra/workflow/envs) but most of the dependencies are not pinned.

Awesome, I think that's the move right there. Basically finding which tool uses the deprecated Bokeh and pinning the 2.4.3 version. Same with pandas and finding out which tool is still using iteritems for pd.Series and pinning the 1.5.3 version.

[411an13]

@411an13 Was this with the PyPI version of Qadabra? Installing directly from GitHub should, I think, resolve these. I'll try to get an updated version on PyPI soon.

Yes, this was with the version installed using pip/PyPI

[gibsramen]

Great, thanks for confirming. I'll try to get a new release out in the next couple of days.

[411an13]

Thank you!

[gibsramen]

v0.3.0 should fix this. Closing for now but feel free to reopen if the issue persists.

[411an13]

Thank you. I tried again with the latest version from PyPi and the workflow ran completely without erroring out.

[yangchen2]

Thanks Allan and Josh, and Gibs for your help!