Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Append distance based QC fails to qcreport #233

Closed
jdaeth274 opened this issue Oct 17, 2022 · 4 comments
Closed

Append distance based QC fails to qcreport #233

jdaeth274 opened this issue Oct 17, 2022 · 4 comments

Comments

@jdaeth274
Copy link

jdaeth274 commented Oct 17, 2022

Hi guys

More of a feature request. Would it be possible to append isolate ids that fail QC based on max-pi-dist and max-a-dist to the output qcreport.txt file with other isolates that fail length and Ns QC? Currently, the ids for these distance-based outliers are just printed to the screen after distance calculation in --create-db mode.

That's for me running:

  • PopPUNK v2.4.5
  • pp-sketchlib v2.0.0
@johnlees
Copy link
Member

Would it be possible to give it a go with v2.5.0 of PopPUNK? We re-did the QC command: https://poppunk.readthedocs.io/en/latest/qc.html

If that's not doing what you want, let us know and we'll add this!

@jdaeth274
Copy link
Author

Thanks John

Sorry I'm just running into an error now when I've upgraded to v2.5.0. Seems to be from HDBScan, not sure if this is related to issue #213.

Error message

$ poppunk --create-db --r-files cdip_genbank_refseq_ukhsa_836_pop.tsv --output cdip_836_poppunk_2.5 --max-pi-dist 0.03 
Traceback (most recent call last):
  File "/home/phe.gov.uk/joshua.daeth/anaconda3/envs/poppunk_env/bin/poppunk", line 11, in <module>
    sys.exit(main())
  File "/home/phe.gov.uk/joshua.daeth/anaconda3/envs/poppunk_env/lib/python3.10/site-packages/PopPUNK/__main__.py", line 213, in main
    from .models import loadClusterFit, BGMMFit, DBSCANFit, RefineFit, LineageFit
  File "/home/phe.gov.uk/joshua.daeth/anaconda3/envs/poppunk_env/lib/python3.10/site-packages/PopPUNK/models.py", line 19, in <module>
    import hdbscan
  File "/home/phe.gov.uk/joshua.daeth/anaconda3/envs/poppunk_env/lib/python3.10/site-packages/hdbscan/__init__.py", line 1, in <module>
    from .hdbscan_ import HDBSCAN, hdbscan
  File "/home/phe.gov.uk/joshua.daeth/anaconda3/envs/poppunk_env/lib/python3.10/site-packages/hdbscan/hdbscan_.py", line 509, in <module>
    memory=Memory(cachedir=None, verbose=0),
TypeError: Memory.__init__() got an unexpected keyword argument 'cachedir'

Versions

  • PopPUNK v2.5.0
  • ppsketchlib v2.0.0

Conda list

# packages in environment at /home/phe.gov.uk/joshua.daeth/anaconda3/envs/poppunk_env:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
aom                       3.5.0                h27087fc_0    conda-forge
at-spi2-atk               2.38.0               h0630a04_3    conda-forge
at-spi2-core              2.40.3               h0630a04_0    conda-forge
atk-1.0                   2.36.0               h3371d22_4    conda-forge
biopython                 1.79            py310h5764c6d_2    conda-forge
boost                     1.74.0          py310h7c3ba0c_5    conda-forge
boost-cpp                 1.74.0               h75c5d50_8    conda-forge
brotli                    1.0.9                h166bdaf_7    conda-forge
brotli-bin                1.0.9                h166bdaf_7    conda-forge
brotlipy                  0.7.0           py310h5764c6d_1004    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2022.9.24            ha878542_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cairo                     1.16.0            ha61ee94_1014    conda-forge
cairomm                   1.14.4               ha770c72_0    conda-forge
cairomm-1.0               1.14.4               h09cb3b9_0    conda-forge
certifi                   2022.9.24          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310h255011f_0    conda-forge
charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
colorama                  0.4.5              pyhd8ed1ab_0    conda-forge
contourpy                 1.0.5           py310hbf28c38_0    conda-forge
cryptography              37.0.1          py310h9ce1e76_0  
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
dendropy                  4.5.2              pyh3252c3a_0    bioconda
docopt                    0.6.2                      py_1    conda-forge
epoxy                     1.5.10               h166bdaf_1    conda-forge
expat                     2.4.9                h27087fc_0    conda-forge
ffmpeg                    5.1.2           gpl_he10e716_101    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.0               hc2a2eb6_1    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.37.4          py310h5764c6d_0    conda-forge
freetype                  2.12.1               hca18f0e_0    conda-forge
fribidi                   1.0.10               h36c2ea0_0    conda-forge
gdk-pixbuf                2.42.8               hff1cb4f_1    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
glib-tools                2.74.0               h6239696_0    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
gnutls                    3.7.8                hf3e180e_0    conda-forge
graph-tool                2.45            py310haee70ea_2    conda-forge
graph-tool-base           2.45            py310hd8094d8_2    conda-forge
graphite2                 1.3.13            h58526e2_1001    conda-forge
gtk3                      3.24.34              h4d20fae_1    conda-forge
h5py                      3.7.0           nompi_py310h416281c_101    conda-forge
harfbuzz                  5.3.0                h418a68e_0    conda-forge
hdbscan                   0.8.28          py310h96516ba_1    conda-forge
hdf5                      1.12.2          nompi_h4df4325_100    conda-forge
hicolor-icon-theme        0.17                 ha770c72_2    conda-forge
icu                       70.1                 h27087fc_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
joblib                    1.2.0              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h166bdaf_2    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.4           py310hbf28c38_0    conda-forge
krb5                      1.19.3               h08a2579_0    conda-forge
lame                      3.100             h166bdaf_1003    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.39                 hc81fddc_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libblas                   3.9.0           16_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h166bdaf_7    conda-forge
libbrotlidec              1.0.9                h166bdaf_7    conda-forge
libbrotlienc              1.0.9                h166bdaf_7    conda-forge
libcblas                  3.9.0           16_linux64_openblas    conda-forge
libcups                   2.3.3                h3e49a29_2    conda-forge
libcurl                   7.85.0               h2283fc2_0    conda-forge
libdeflate                1.14                 h166bdaf_0    conda-forge
libdrm                    2.4.113              h166bdaf_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.2.0              h65d4601_18    conda-forge
libgfortran-ng            12.2.0              h69a702a_18    conda-forge
libgfortran5              12.2.0              h337968e_18    conda-forge
libgirepository           1.74.0               ha2a38d2_0    conda-forge
libglib                   2.74.0               h7a41b64_0    conda-forge
libgomp                   12.2.0              h65d4601_18    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libidn2                   2.3.3                h166bdaf_0    conda-forge
liblapack                 3.9.0           16_linux64_openblas    conda-forge
libnghttp2                1.47.0               hff17c54_1    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.21          pthreads_h78a6416_3    conda-forge
libpciaccess              0.16                 h516909a_0    conda-forge
libpng                    1.6.38               h753d276_0    conda-forge
librsvg                   2.54.4               h7abd40a_0    conda-forge
libsqlite                 3.39.4               h753d276_0    conda-forge
libssh2                   1.10.0               hf14f497_3    conda-forge
libstdcxx-ng              12.2.0              h46fd767_18    conda-forge
libtasn1                  4.19.0               h166bdaf_0    conda-forge
libtiff                   4.4.0                h55922b4_4    conda-forge
libunistring              0.9.10               h7f98852_0    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libva                     2.16.0               h166bdaf_0    conda-forge
libvpx                    1.11.0               h9c3ff4c_3    conda-forge
libwebp-base              1.2.4                h166bdaf_0    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxml2                   2.9.14               h22db469_4    conda-forge
libzlib                   1.2.13               h166bdaf_4    conda-forge
mandrake                  1.2.2           py310h7dbff7e_1    conda-forge
matplotlib-base           3.6.1           py310h8d5ebf3_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
nettle                    3.8.1                hc379101_1    conda-forge
networkx                  2.8.7              pyhd8ed1ab_0    conda-forge
numpy                     1.23.3          py310h53a5b5f_0    conda-forge
openblas                  0.3.21          pthreads_h320a7e8_3    conda-forge
openh264                  2.3.1                h27087fc_1    conda-forge
openjpeg                  2.5.0                h7d73246_1    conda-forge
openssl                   3.0.5                h166bdaf_2    conda-forge
p11-kit                   0.24.1               hc5aa10d_0    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandas                    1.5.0           py310h769672d_0    conda-forge
pango                     1.50.11              h382ae3d_0    conda-forge
pcre2                     10.37                hc3806b6_1    conda-forge
pillow                    9.2.0           py310hbd86126_2    conda-forge
pip                       22.3               pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
plotly                    5.10.0             pyhd8ed1ab_0    conda-forge
poppunk                   2.5.0           py310h2579afa_0    bioconda
pp-sketchlib              2.0.0           py310h5a37817_2    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pycairo                   1.21.0          py310h96fc21a_1    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygobject                 3.42.2          py310h964465f_0    conda-forge
pyopenssl                 22.0.0             pyhd8ed1ab_1    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.6          ha86cf86_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.10                    2_cp310    conda-forge
pytz                      2022.4             pyhd8ed1ab_0    conda-forge
rapidnj                   2.3.2                h9f5acd7_2    bioconda
readline                  8.1.2                h0f457ee_0    conda-forge
requests                  2.28.1             pyhd8ed1ab_1    conda-forge
scikit-learn              1.1.2           py310h0c3af53_0    conda-forge
scipy                     1.9.1           py310hdfbd76f_0    conda-forge
setuptools                65.5.0             pyhd8ed1ab_0    conda-forge
sigcpp-2.0                2.10.8               h27087fc_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sparsehash                2.0.4                h9c3ff4c_0    conda-forge
svt-av1                   1.2.1                h27087fc_0    conda-forge
tenacity                  8.1.0              pyhd8ed1ab_0    conda-forge
threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tqdm                      4.64.1             pyhd8ed1ab_0    conda-forge
treeswift                 1.1.28             pyh5e36f6f_0    bioconda
tzdata                    2022e                h191b570_0    conda-forge
unicodedata2              14.0.0          py310h5764c6d_1    conda-forge
urllib3                   1.26.11            pyhd8ed1ab_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
x264                      1!164.3095           h166bdaf_2    conda-forge
x265                      3.5                  h924138e_3    conda-forge
xorg-compositeproto       0.4.2             h7f98852_1001    conda-forge
xorg-damageproto          1.2.1             h7f98852_1002    conda-forge
xorg-fixesproto           5.0               h7f98852_1002    conda-forge
xorg-inputproto           2.3.2             h7f98852_1002    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.0.10               h7f98852_0    conda-forge
xorg-libsm                1.2.3             hd9c2040_1000    conda-forge
xorg-libx11               1.6.12               h36c2ea0_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxaw               1.0.14               h7f98852_0    conda-forge
xorg-libxcomposite        0.4.5                h7f98852_0    conda-forge
xorg-libxcursor           1.2.0                h516909a_0    conda-forge
xorg-libxdamage           1.1.5                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h516909a_0    conda-forge
xorg-libxfixes            5.0.3             h516909a_1004    conda-forge
xorg-libxi                1.7.10               h516909a_0    conda-forge
xorg-libxinerama          1.1.4             h9c3ff4c_1001    conda-forge
xorg-libxmu               1.1.3                h516909a_0    conda-forge
xorg-libxpm               3.5.13               h516909a_0    conda-forge
xorg-libxrandr            1.5.2                h516909a_1    conda-forge
xorg-libxrender           0.9.10            h516909a_1002    conda-forge
xorg-libxt                1.1.5             h516909a_1003    conda-forge
xorg-libxtst              1.2.3             h516909a_1002    conda-forge
xorg-randrproto           1.5.0             h7f98852_1001    conda-forge
xorg-recordproto          1.14.2            h7f98852_1002    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-util-macros          1.19.3               h7f98852_0    conda-forge
xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zlib                      1.2.13               h166bdaf_4    conda-forge
zstandard                 0.18.0          py310h5764c6d_0    conda-forge
zstd                      1.5.2                h6239696_4    conda-forge

@nickjcroucher
Copy link
Collaborator

conda install "joblib<=1.1" should fix that

@jdaeth274
Copy link
Author

Thanks Nick

That now works, the updated QC output is good too.
Cheers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants