Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] cudaErrorIllegalAddress while computing ranking... #9703

Closed
rilango opened this issue Nov 16, 2021 · 11 comments · Fixed by #9778
Closed

[BUG] cudaErrorIllegalAddress while computing ranking... #9703

rilango opened this issue Nov 16, 2021 · 11 comments · Fixed by #9778
Assignees
Labels
bug Something isn't working Python Affects Python cuDF API.

Comments

@rilango
Copy link

rilango commented Nov 16, 2021

Describe the bug
While attempting to compute rank(cudf.rank), we face the following error

RuntimeError                              Traceback (most recent call last)
<ipython-input-3-09f0e5240771> in <module>
----> 1 df.rank()

/opt/conda/lib/python3.8/site-packages/cudf/core/frame.py in rank(self, axis, method, numeric_only, na_option, ascending, pct)
   1508                 return source.astype("float64")
   1509 
-> 1510         data, index = libcudf.sort.rank_columns(
   1511             source, method_enum, na_option, ascending, pct
   1512         )

cudf/_lib/sort.pyx in cudf._lib.sort.rank_columns()

RuntimeError: reduce_by_key: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered

Steps/Code to reproduce bug
Data file is at https://drive.google.com/file/d/1-hIshmdiYd70BhDnfgYk-BSF89JU3HiG/view?usp=sharing

import cudf
df = cudf.read_csv('/data/forranking.csv')
df.rank()

Expected behavior
Return the dataframe with ranks

Environment overview (please complete the following information)

  • Environment location: Docker
  • Method of cuDF install: conda

Environment details

Click here to see environment details
 **git***
 commit 08fa1b9f113bd384d30845ca4a01e4ab78d2417b (HEAD -> rilango/bm_refactor, origin/rilango/bm_refactor)
 Author: Rajesh K Ilango <[email protected]>
 Date:   Tue Nov 16 11:37:58 2021 -0800
 
 Temporary move Ranking computation to CPU.
 
 CUDF version of ranking computation works for up to 1000 molecules,
 crashing on upto 3000 and above which it will freeze.
 **git submodules***
 
 ***OS Information***
 DISTRIB_ID=Ubuntu
 DISTRIB_RELEASE=20.04
 DISTRIB_CODENAME=focal
 DISTRIB_DESCRIPTION="Ubuntu 20.04.3 LTS"
 NAME="Ubuntu"
 VERSION="20.04.3 LTS (Focal Fossa)"
 ID=ubuntu
 ID_LIKE=debian
 PRETTY_NAME="Ubuntu 20.04.3 LTS"
 VERSION_ID="20.04"
 HOME_URL="https://www.ubuntu.com/"
 SUPPORT_URL="https://help.ubuntu.com/"
 BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
 PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
 VERSION_CODENAME=focal
 UBUNTU_CODENAME=focal
 Linux rilango-dt1 5.11.0-40-generic #44~20.04.2-Ubuntu SMP Tue Oct 26 18:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
 
 ***GPU Information***
 Tue Nov 16 20:17:08 2021
 +-----------------------------------------------------------------------------+
 | NVIDIA-SMI 470.82.01    Driver Version: 470.82.01    CUDA Version: 11.4     |
 |-------------------------------+----------------------+----------------------+
 | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
 | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
 |                               |                      |               MIG M. |
 |===============================+======================+======================|
 |   0  NVIDIA GeForce ...  Off  | 00000000:68:00.0 Off |                  N/A |
 | 30%   47C    P8    25W / 350W |     19MiB / 24265MiB |      0%      Default |
 |                               |                      |                  N/A |
 +-------------------------------+----------------------+----------------------+
 
 +-----------------------------------------------------------------------------+
 | Processes:                                                                  |
 |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
 |        ID   ID                                                   Usage      |
 |=============================================================================|
 +-----------------------------------------------------------------------------+
 
 ***CPU***
 Architecture:                    x86_64
 CPU op-mode(s):                  32-bit, 64-bit
 Byte Order:                      Little Endian
 Address sizes:                   46 bits physical, 48 bits virtual
 CPU(s):                          20
 On-line CPU(s) list:             0-19
 Thread(s) per core:              2
 Core(s) per socket:              10
 Socket(s):                       1
 NUMA node(s):                    1
 Vendor ID:                       GenuineIntel
 CPU family:                      6
 Model:                           85
 Model name:                      Intel(R) Core(TM) i9-9820X CPU @ 3.30GHz
 Stepping:                        4
 CPU MHz:                         3300.000
 CPU max MHz:                     4200.0000
 CPU min MHz:                     1200.0000
 BogoMIPS:                        6599.98
 Virtualization:                  VT-x
 L1d cache:                       320 KiB
 L1i cache:                       320 KiB
 L2 cache:                        10 MiB
 L3 cache:                        16.5 MiB
 NUMA node0 CPU(s):               0-19
 Vulnerability Itlb multihit:     KVM: Mitigation: VMX disabled
 Vulnerability L1tf:              Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
 Vulnerability Mds:               Mitigation; Clear CPU buffers; SMT vulnerable
 Vulnerability Meltdown:          Mitigation; PTI
 Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
 Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
 Vulnerability Spectre v2:        Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
 Vulnerability Srbds:             Not affected
 Vulnerability Tsx async abort:   Mitigation; Clear CPU buffers; SMT vulnerable
 Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req md_clear flush_l1d
 
 ***CMake***
 /opt/conda/bin/cmake
 cmake version 3.20.4
 
 CMake suite maintained and supported by Kitware (kitware.com/cmake).
 
 ***g++***
 /usr/bin/g++
 g++ (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
 Copyright (C) 2019 Free Software Foundation, Inc.
 This is free software; see the source for copying conditions.  There is NO
 warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 
 
 ***nvcc***
 /usr/local/cuda/bin/nvcc
 nvcc: NVIDIA (R) Cuda compiler driver
 Copyright (c) 2005-2021 NVIDIA Corporation
 Built on Mon_May__3_19:15:13_PDT_2021
 Cuda compilation tools, release 11.3, V11.3.109
 Build cuda_11.3.r11.3/compiler.29920130_0
 
 ***Python***
 /opt/conda/bin/python
 Python 3.8.10
 
 ***Environment Variables***
 PATH                            : /usr/local/nvm/versions/node/v15.12.0/bin:/opt/conda/bin:/opt/cmake-3.14.6-Linux-x86_64/bin/:/usr/local/mpi/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/ucx/bin:/opt/tensorrt/bin
 LD_LIBRARY_PATH                 : /usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
 NUMBAPRO_NVVM                   :
 NUMBAPRO_LIBDEVICE              :
 CONDA_PREFIX                    :
 PYTHON_PATH                     :
 
 ***conda packages***
 /opt/conda/bin/conda
 # packages in environment at /opt/conda:
 #
 # Name                    Version                   Build  Channel
 _libgcc_mutex             0.1                 conda_forge    conda-forge
 _openmp_mutex             4.5                      1_llvm    conda-forge
 abseil-cpp                20210324.2           h2531618_0
 absl-py                   0.13.0                   pypi_0    pypi
 aiohttp                   3.7.4.post0      py38h497a2fe_0    conda-forge
 alabaster                 0.7.12                   pypi_0    pypi
 antlr4-python3-runtime    4.8                      pypi_0    pypi
 anyio                     2.2.0            py38h06a4308_1
 apex                      0.1                      pypi_0    pypi
 appdirs                   1.4.4                    pypi_0    pypi
 argon2-cffi               20.1.0           py38h27cfd23_1
 arrow-cpp                 5.0.0           py38hfd64638_1_cuda    conda-forge
 arrow-cpp-proc            3.0.0                      cuda    conda-forge
 ascii-graph               1.5.1                    pypi_0    pypi
 asgiref                   3.3.4                    pypi_0    pypi
 async-generator           1.10                     pypi_0    pypi
 async-timeout             3.0.1                   py_1000    conda-forge
 atk-1.0                   2.36.0               h3371d22_4    conda-forge
 attrdict                  2.0.1                    pypi_0    pypi
 attrs                     21.2.0             pyhd8ed1ab_0    conda-forge
 audioread                 2.1.9                    pypi_0    pypi
 aws-c-cal                 0.5.11               h95a6274_0    conda-forge
 aws-c-common              0.6.2                h27cfd23_0
 aws-c-event-stream        0.2.7               h3541f99_13    conda-forge
 aws-c-io                  0.10.5               hfb6a706_0    conda-forge
 aws-checksums             0.1.11               ha31a3da_7    conda-forge
 aws-sdk-cpp               1.8.186              hb4091e7_3    conda-forge
 babel                     2.9.1                    pypi_0    pypi
 backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
 backports                 1.0                        py_2    conda-forge
 backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
 beautifulsoup4            4.9.3              pyhb0f4dca_0    conda-forge
 black                     19.10b0                  pypi_0    pypi
 blas                      2.109                  openblas    conda-forge
 blas-devel                3.9.0                9_openblas    conda-forge
 bleach                    3.3.0                    pypi_0    pypi
 blosc                     1.21.0               h9c3ff4c_0    conda-forge
 bokeh                     2.3.3            py38h06a4308_0
 boost                     1.74.0           py38hc10631b_3    conda-forge
 boost-cpp                 1.74.0               h312852a_4    conda-forge
 boto                      2.49.0                     py_0    conda-forge
 boto3                     1.17.96            pyhd8ed1ab_0    conda-forge
 botocore                  1.20.96            pyhd8ed1ab_0    conda-forge
 bottleneck                1.3.2            py38heb32a55_1
 braceexpand               0.1.7                    pypi_0    pypi
 brotli                    1.0.9                he6710b0_2
 brotlipy                  0.7.0           py38h497a2fe_1001    conda-forge
 brunsli                   0.1                  h2531618_0
 bz2file                   0.98                       py_0    conda-forge
 bzip2                     1.0.8                h7f98852_4    conda-forge
 c-ares                    1.17.1               h7f98852_1    conda-forge
 ca-certificates           2021.10.26           h06a4308_2
 cachetools                4.2.2              pyhd8ed1ab_0    conda-forge
 cairo                     1.16.0            h6cf1ce9_1008    conda-forge
 catalogue                 2.0.4            py38h578d9bd_0    conda-forge
 certifi                   2021.10.8        py38h06a4308_0
 cffi                      1.14.5           py38ha65f79e_0    conda-forge
 cfitsio                   3.470                hb418390_7    conda-forge
 chardet                   4.0.0            py38h578d9bd_1    conda-forge
 charls                    2.2.0                h2531618_0
 click                     7.1.2              pyh9f0ad1d_0    conda-forge
 click-plugins             1.1.1              pyhd3eb1b0_0
 cligj                     0.7.2            py38h06a4308_0
 cloudpickle               2.0.0              pyhd3eb1b0_0
 cmake                     3.20.4               h8897547_0    conda-forge
 codecov                   2.1.11                   pypi_0    pypi
 colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
 colorcet                  2.0.6              pyhd3eb1b0_0
 conda                     4.10.1           py38h06a4308_1
 conda-build               3.21.4           py38h578d9bd_0    conda-forge
 conda-package-handling    1.7.3            py38h497a2fe_0    conda-forge
 configparser              5.0.2                    pypi_0    pypi
 coverage                  5.5                      pypi_0    pypi
 cryptography              3.4.7            py38ha5dfef3_0    conda-forge
 cuchem-commons            0.1                      pypi_0    pypi
 cucim                     21.10.00        cuda_11.2_py38_gd7ac21f_0    rapidsai
 cudatoolkit               11.2.72              h2bc3f7f_0    nvidia
 cudf                      21.10.01        cuda_11.2_py38_ga1d2d13a14_0    rapidsai
 cudf_kafka                21.10.01        py38_ga1d2d13a14_0    rapidsai
 cugraph                   21.10.00        cuda11.2_py38_g84617024_0    rapidsai
 cuml                      21.10.00        cuda11.2_py38_g0fd3503ba_0    rapidsai
 cupy                      9.0.0            py38ha69542f_0    conda-forge
 curl                      7.77.0               hea6ffbf_0    conda-forge
 cusignal                  21.10.00        py37_gff14a10_0    rapidsai
 cuspatial                 21.10.00        py38_gba20298_0    rapidsai
 custreamz                 21.10.01        py38_ga1d2d13a14_0    rapidsai
 cuxfilter                 21.10.00        py38_g003d3d6_0    rapidsai
 cycler                    0.10.0                   pypi_0    pypi
 cymem                     2.0.5            py38h709712a_1    conda-forge
 cyrus-sasl                2.1.27               h758a394_8
 cython                    0.28.4                   pypi_0    pypi
 cython-blis               0.7.4            py38h5c078b8_0    conda-forge
 cytoolz                   0.11.0           py38h7b6447c_0
 dask                      2021.8.1                 pypi_0    pypi
 dask-cuda                 21.10.00                 py38_0    rapidsai
 dask-cudf                 21.10.01        py38_ga1d2d13a14_0    rapidsai
 dataclasses               0.8                pyhc8e2a94_1    conda-forge
 dataproperty              0.50.1                   pypi_0    pypi
 datashader                0.11.1                     py_0
 datashape                 0.5.4            py38h06a4308_1
 debugpy                   1.4.1            py38h295c915_0
 decorator                 5.0.9              pyhd8ed1ab_0    conda-forge
 deepspeed                 0.3.10                   pypi_0    pypi
 defusedxml                0.7.1              pyhd3eb1b0_0
 distance                  0.1.3                    pypi_0    pypi
 distributed               2021.8.1                 pypi_0    pypi
 django                    3.2.5                    pypi_0    pypi
 dllogger                  0.1.0                    pypi_0    pypi
 dlpack                    0.5                  h9c3ff4c_0    conda-forge
 docker-pycreds            0.4.0                    pypi_0    pypi
 docopt                    0.6.2                    pypi_0    pypi
 docutils                  0.16                     pypi_0    pypi
 editdistance              0.5.3                    pypi_0    pypi
 entrypoints               0.3                      pypi_0    pypi
 expat                     2.4.1                h9c3ff4c_0    conda-forge
 faiss-proc                1.0.0                      cuda    rapidsai
 fastavro                  1.4.3            py38h7f8727e_0
 fastrlock                 0.6              py38h2531618_0
 filelock                  3.0.12             pyh9f0ad1d_0    conda-forge
 fiona                     1.8.20           py38ha695d3a_1    conda-forge
 flake8                    3.7.9                    pypi_0    pypi
 flask                     2.0.1                    pypi_0    pypi
 flatbuffers               2.0                      pypi_0    pypi
 fmt                       7.1.3                hff7bd54_0
 font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
 font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
 font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
 font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
 fontconfig                2.13.1            hba837de_1005    conda-forge
 fonts-conda-ecosystem     1                             0    conda-forge
 fonts-conda-forge         1                             0    conda-forge
 fonttools                 4.25.0             pyhd3eb1b0_0
 freetype                  2.10.4               h0708190_1    conda-forge
 freexl                    1.0.6                h27cfd23_0
 fribidi                   1.0.10               h36c2ea0_0    conda-forge
 frozendict                2.0.3                    pypi_0    pypi
 fsspec                    2021.7.0                 pypi_0    pypi
 future                    0.18.2                   pypi_0    pypi
 g2p-en                    2.1.0                    pypi_0    pypi
 gdal                      3.3.0            py38h507a4fd_3    conda-forge
 gdk-pixbuf                2.42.6               h04a7f16_0    conda-forge
 gdown                     3.13.0                   pypi_0    pypi
 geopandas                 0.9.0                      py_1
 geopandas-base            0.9.0                      py_1
 geos                      3.9.1                h2531618_0
 geotiff                   1.6.0                h4f31c25_6    conda-forge
 gettext                   0.19.8.1          h0b5b191_1005    conda-forge
 gflags                    2.2.2                he6710b0_0
 giflib                    5.2.1                h36c2ea0_2    conda-forge
 gitdb                     4.0.7                    pypi_0    pypi
 gitpython                 3.1.18                   pypi_0    pypi
 glob2                     0.7                        py_0    conda-forge
 glog                      0.5.0                h2531618_0
 google-api-core           1.30.0             pyhd8ed1ab_0    conda-forge
 google-auth               1.30.2             pyh6c4a22f_0    conda-forge
 google-auth-oauthlib      0.4.4                    pypi_0    pypi
 google-cloud-core         1.7.0              pyh6c4a22f_0    conda-forge
 google-cloud-storage      1.38.0             pyh44b312d_0    conda-forge
 google-crc32c             1.1.2            py38h8838a9a_0    conda-forge
 google-resumable-media    1.3.0              pyh6c4a22f_0    conda-forge
 googleapis-common-protos  1.53.0           py38h578d9bd_0    conda-forge
 graphite2                 1.3.13            h58526e2_1001    conda-forge
 graphql-core              3.1.5                    pypi_0    pypi
 graphsurgeon              0.4.5                    pypi_0    pypi
 graphviz                  2.48.0               h85b4f2f_0    conda-forge
 grpc-cpp                  1.39.0               h36ce80c_1    conda-forge
 grpcio                    1.38.0           py38hdd6454d_0    conda-forge
 gtk2                      2.24.33              h539f30e_1    conda-forge
 gts                       0.7.6                h64030ff_2    conda-forge
 gunicorn                  20.1.0                   pypi_0    pypi
 h11                       0.12.0                   pypi_0    pypi
 h5py                      3.2.1                    pypi_0    pypi
 harfbuzz                  2.8.2                h83ec7ef_0    conda-forge
 hdf4                      4.2.15               h10796ff_3    conda-forge
 hdf5                      1.10.6          nompi_h6a2412b_1114    conda-forge
 heapdict                  1.0.1              pyhd3eb1b0_0
 html2text                 2020.1.16                pypi_0    pypi
 httptools                 0.2.0                    pypi_0    pypi
 huggingface-hub           0.0.12                   pypi_0    pypi
 hydra-core                1.1.0                    pypi_0    pypi
 hypothesis                4.50.8                   pypi_0    pypi
 icu                       68.1                 h58526e2_0    conda-forge
 idna                      2.10               pyh9f0ad1d_0    conda-forge
 imagecodecs               2021.3.31        py38hbfaf385_1    conda-forge
 imageio                   2.9.0              pyhd3eb1b0_0
 imagesize                 1.2.0                    pypi_0    pypi
 importlib-metadata        4.8.1            py38h06a4308_0
 importlib-resources       5.2.2                    pypi_0    pypi
 importlib_metadata        4.8.1                hd3eb1b0_0
 inflect                   5.3.0                    pypi_0    pypi
 iniconfig                 1.1.1                    pypi_0    pypi
 ipadic                    1.0.0                    pypi_0    pypi
 ipdb                      0.13.9                   pypi_0    pypi
 ipykernel                 5.5.5                    pypi_0    pypi
 ipython                   7.24.1           py38hd0cf306_0    conda-forge
 ipython_genutils          0.2.0                      py_1    conda-forge
 ipywidgets                7.6.3                    pypi_0    pypi
 isort                     4.3.21                   pypi_0    pypi
 itsdangerous              2.0.1                    pypi_0    pypi
 jbig                      2.1               h7f98852_2003    conda-forge
 jedi                      0.18.0           py38h578d9bd_2    conda-forge
 jieba                     0.42.1                   pypi_0    pypi
 jinja2                    3.0.1              pyhd8ed1ab_0    conda-forge
 jmespath                  0.10.0             pyh9f0ad1d_0    conda-forge
 joblib                    1.0.1                    pypi_0    pypi
 jpeg                      9d                   h36c2ea0_0    conda-forge
 json-c                    0.15                 h98cffda_0    conda-forge
 json5                     0.9.5                    pypi_0    pypi
 jsonschema                3.2.0              pyhd3eb1b0_2
 jupyter-client            6.1.12                   pypi_0    pypi
 jupyter-core              4.7.1                    pypi_0    pypi
 jupyter-server-proxy      3.1.0              pyhd8ed1ab_0    conda-forge
 jupyter-tensorboard       0.2.0                    pypi_0    pypi
 jupyter_client            7.0.1              pyhd3eb1b0_0
 jupyter_core              4.8.1            py38h06a4308_0
 jupyter_server            1.4.1            py38h06a4308_0
 jupyterlab                2.3.1                    pypi_0    pypi
 jupyterlab-pygments       0.1.2                    pypi_0    pypi
 jupyterlab-server         1.2.0                    pypi_0    pypi
 jupyterlab_widgets        1.0.0              pyhd3eb1b0_1
 jupytext                  1.11.3                   pypi_0    pypi
 jxrlib                    1.1                  h7b6447c_2
 kaldi-io                  0.9.4                    pypi_0    pypi
 kaldi-python-io           1.2.2                    pypi_0    pypi
 kaldiio                   2.17.2                   pypi_0    pypi
 kealib                    1.4.14               h54c064f_0
 kiwisolver                1.3.1            py38h2531618_0
 krb5                      1.19.1               hcc1bbae_0    conda-forge
 latexcodec                2.0.1                    pypi_0    pypi
 lcms2                     2.12                 h3be6417_0
 ld_impl_linux-64          2.35.1               hea4e1c9_2    conda-forge
 lerc                      2.2.1                h9c3ff4c_0    conda-forge
 libaec                    1.0.4                he6710b0_1
 libarchive                3.5.1                hccf745f_2    conda-forge
 libblas                   3.9.0                9_openblas    conda-forge
 libbrotlicommon           1.0.9                h7f98852_5    conda-forge
 libbrotlidec              1.0.9                h7f98852_5    conda-forge
 libbrotlienc              1.0.9                h7f98852_5    conda-forge
 libcblas                  3.9.0                9_openblas    conda-forge
 libcrc32c                 1.1.1                h9c3ff4c_2    conda-forge
 libcucim                  21.10.00        cuda11.2_gd7ac21f_0    rapidsai
 libcudf                   21.10.01        cuda11.2_ga1d2d13a14_0    rapidsai
 libcudf_kafka             21.10.01          ga1d2d13a14_0    rapidsai
 libcugraph                21.10.00        cuda11.2_g84617024_0    rapidsai
 libcuml                   21.10.00        cuda11.2_g0fd3503ba_0    rapidsai
 libcumlprims              21.10.00        cuda11.2_g167dc59_0    nvidia
 libcurl                   7.77.0               h2574ce0_0    conda-forge
 libcuspatial              21.10.00        cuda11.2_gba20298_0    rapidsai
 libdap4                   3.20.6               hd7c4107_2    conda-forge
 libdb                     6.2.32               hf484d3e_0
 libdeflate                1.7                  h7f98852_5    conda-forge
 libedit                   3.1.20191231         he28a2e2_2    conda-forge
 libev                     4.33                 h516909a_1    conda-forge
 libevent                  2.1.8                h1ba5d50_1
 libfaiss                  1.7.0           cuda112h5bea7ad_8_cuda    conda-forge
 libffi                    3.3                  h58526e2_2    conda-forge
 libgcc-ng                 9.3.0               h2828fa1_19    conda-forge
 libgcrypt                 1.9.3                h27cfd23_0
 libgd                     2.3.2                h78a0170_0    conda-forge
 libgdal                   3.3.0                h8f005ca_3    conda-forge
 libgfortran-ng            9.3.0               hff62375_19    conda-forge
 libgfortran5              9.3.0               hff62375_19    conda-forge
 libglib                   2.68.3               h3e27bee_0    conda-forge
 libgomp                   9.3.0               h2828fa1_19    conda-forge
 libgpg-error              1.42                 h2531618_0
 libgsasl                  1.8.0                         2    conda-forge
 libhwloc                  2.3.0                h5e5b7d1_1    conda-forge
 libiconv                  1.16                 h516909a_0    conda-forge
 libkml                    1.3.0             h238a007_1014    conda-forge
 liblapack                 3.9.0                9_openblas    conda-forge
 liblapacke                3.9.0                9_openblas    conda-forge
 liblief                   0.11.5               h9c3ff4c_0    conda-forge
 libllvm10                 10.0.1               he513fc3_3    conda-forge
 libnetcdf                 4.8.0           nompi_hcd642e3_103    conda-forge
 libnghttp2                1.43.0               h812cca2_0    conda-forge
 libntlm                   1.6                  h7f8727e_0
 libopenblas               0.3.15          pthreads_h8fe5266_1    conda-forge
 libpng                    1.6.37               h21135ba_2    conda-forge
 libpq                     13.3                 hd57d9b9_0    conda-forge
 libprotobuf               3.16.0               h780b84a_0    conda-forge
 librdkafka                1.6.1                hc49e61c_1    conda-forge
 librmm                    21.10.01             he245796_0    conda-forge
 librosa                   0.8.0                    pypi_0    pypi
 librsvg                   2.50.7               hc3c00ef_0    conda-forge
 librttopo                 1.1.0                h1185371_6    conda-forge
 libsodium                 1.0.18               h7b6447c_0
 libspatialindex           1.9.3                h2531618_0
 libspatialite             5.0.1                h8694cbe_5    conda-forge
 libssh2                   1.9.0                ha56f1ee_6    conda-forge
 libstdcxx-ng              9.3.0               h6de172a_19    conda-forge
 libthrift                 0.14.2               hcc01f38_0
 libtiff                   4.3.0                hf544144_1    conda-forge
 libtool                   2.4.6             h58526e2_1007    conda-forge
 libutf8proc               2.6.1                h27cfd23_0
 libuuid                   2.32.1            h7f98852_1000    conda-forge
 libuv                     1.41.0               h7f98852_0    conda-forge
 libwebp                   1.2.0                h3452ae3_0    conda-forge
 libwebp-base              1.2.0                h7f98852_2    conda-forge
 libxcb                    1.13              h7f98852_1003    conda-forge
 libxgboost                1.4.2dev.rapidsai21.10      cuda11.2_0    rapidsai
 libxml2                   2.9.12               h72842e0_0    conda-forge
 libzip                    1.8.0                h4de3113_0    conda-forge
 libzlib                   1.2.11            h36c2ea0_1013    conda-forge
 libzopfli                 1.0.3                he6710b0_0
 llvm-openmp               11.1.0               h4bd325d_1    conda-forge
 llvmlite                  0.36.0           py38hf484d3e_0    numba
 lmdb                      1.2.1                    pypi_0    pypi
 locket                    0.2.1            py38h06a4308_1
 lz4-c                     1.9.3                h9c3ff4c_0    conda-forge
 lzo                       2.10              h516909a_1000    conda-forge
 magma-cuda110             2.5.2                         5    local
 mako                      1.1.4                    pypi_0    pypi
 mapclassify               2.4.3              pyhd3eb1b0_0
 markdown                  3.3.4            py38h06a4308_0
 markdown-it-py            1.1.0                    pypi_0    pypi
 markupsafe                2.0.1            py38h497a2fe_0    conda-forge
 marshmallow               3.13.0                   pypi_0    pypi
 maskrcnn-benchmark        0.1                      pypi_0    pypi
 matplotlib                3.4.2                    pypi_0    pypi
 matplotlib-base           3.4.3            py38hbbc1b5f_0
 matplotlib-inline         0.1.2              pyhd8ed1ab_2    conda-forge
 mbstrdecoder              1.0.1                    pypi_0    pypi
 mccabe                    0.6.1                    pypi_0    pypi
 mdit-py-plugins           0.2.8                    pypi_0    pypi
 mecab-python3             1.0.3                    pypi_0    pypi
 megatron-lm               2.2.0                    pypi_0    pypi
 mistune                   0.8.4           py38h7b6447c_1000
 mkl                       2019.5                      281    conda-forge
 mkl-include               2019.5                      281    conda-forge
 mlperf-compliance         0.0.10                   pypi_0    pypi
 mock                      4.0.3            py38h578d9bd_1    conda-forge
 mpmath                    1.2.1                    pypi_0    pypi
 msgfy                     0.1.0                    pypi_0    pypi
 msgpack-python            1.0.2            py38hff7bd54_1
 multidict                 5.1.0            py38h497a2fe_1    conda-forge
 multipledispatch          0.6.0                    py38_0
 munch                     2.5.0              pyhd3eb1b0_0
 munkres                   1.1.4                      py_0
 murmurhash                1.0.5            py38h709712a_0    conda-forge
 nbclient                  0.5.3                    pypi_0    pypi
 nbconvert                 6.0.7                    pypi_0    pypi
 nbformat                  5.1.3              pyhd3eb1b0_0
 nccl                      2.11.4.1             hdc17891_0    conda-forge
 ncurses                   6.2                  h58526e2_4    conda-forge
 nemo-toolkit              1.2.0                    pypi_0    pypi
 nest-asyncio              1.5.1              pyhd3eb1b0_0
 networkx                  2.0                      pypi_0    pypi
 ninja                     1.10.2.2                 pypi_0    pypi
 nltk                      3.6.2                    pypi_0    pypi
 nodejs                    14.17.1              h92b4a50_1    conda-forge
 notebook                  6.2.0                    pypi_0    pypi
 numba                     0.53.1          np1.11py3.8h04863e7_g97fe221b3_0    numba
 numexpr                   2.7.3            py38h4be448d_1
 numpy                     1.20.0                   pypi_0    pypi
 nvidia-dali-cuda110       1.2.0                    pypi_0    pypi
 nvidia-dlprof-pytorch-nvtx 1.2.0                    pypi_0    pypi
 nvidia-dlprofviewer       1.2.0                    pypi_0    pypi
 nvtx                      0.2.3            py38h497a2fe_0    conda-forge
 oauthlib                  3.1.1                    pypi_0    pypi
 olefile                   0.46               pyhd3eb1b0_0
 omegaconf                 2.1.0                    pypi_0    pypi
 onnx                      1.10.2                   pypi_0    pypi
 onnxruntime               1.8.0                    pypi_0    pypi
 openblas                  0.3.15          pthreads_h4748800_1    conda-forge
 opencc                    1.1.2                    pypi_0    pypi
 openfst                   1.8.0                h0efe328_0    conda-forge
 openjpeg                  2.4.0                h3ad879b_0
 openssl                   1.1.1l               h7f8727e_0
 orc                       1.6.9                h58a87f1_0    conda-forge
 packaging                 20.9               pyh44b312d_0    conda-forge
 pandas                    1.1.4                    pypi_0    pypi
 pandoc                    2.12                 h06a4308_0
 pandocfilters             1.4.3            py38h06a4308_1
 panel                     0.12.1             pyhd3eb1b0_0
 pango                     1.48.7               hb8ff022_0    conda-forge
 pangu                     4.0.6.1                  pypi_0    pypi
 param                     1.11.1             pyhd3eb1b0_0
 parameterized             0.8.1                    pypi_0    pypi
 parquet-cpp               1.5.1                h34088ae_4
 parso                     0.8.2              pyhd8ed1ab_0    conda-forge
 partd                     1.2.0              pyhd3eb1b0_0
 patchelf                  0.11                 he1b5a44_0    conda-forge
 pathspec                  0.9.0                    pypi_0    pypi
 pathtools                 0.1.2                    pypi_0    pypi
 pathvalidate              2.4.1                    pypi_0    pypi
 pathy                     0.5.2              pyhd8ed1ab_0    conda-forge
 pcre                      8.45                 h9c3ff4c_0    conda-forge
 pesq                      0.0.3                    pypi_0    pypi
 pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
 pickleshare               0.7.5                   py_1003    conda-forge
 pillow                    8.3.1                    pypi_0    pypi
 pillow-simd               7.0.0.post3              pypi_0    pypi
 pip                       21.2.4           py38h06a4308_0
 pip-api                   0.0.20                   pypi_0    pypi
 pipreqs                   0.4.10                   pypi_0    pypi
 pixman                    0.40.0               h36c2ea0_0    conda-forge
 pkginfo                   1.7.0              pyhd8ed1ab_0    conda-forge
 pluggy                    0.13.1                   pypi_0    pypi
 polygraphy                0.30.2                   pypi_0    pypi
 pooch                     1.4.0                    pypi_0    pypi
 poppler                   21.03.0              h93df280_0    conda-forge
 poppler-data              0.4.10               h06a4308_0
 portalocker               2.0.0                    pypi_0    pypi
 postgresql                13.3                 h2510834_0    conda-forge
 preshed                   3.0.5            py38h709712a_0    conda-forge
 prettytable               2.1.0                    pypi_0    pypi
 progressbar               2.5                      pypi_0    pypi
 proj                      8.0.1                h277dcde_0    conda-forge
 prometheus_client         0.11.0             pyhd3eb1b0_0
 promise                   2.3                      pypi_0    pypi
 prompt-toolkit            3.0.18             pyha770c72_0    conda-forge
 protobuf                  3.16.0           py38h709712a_0    conda-forge
 psutil                    5.8.0            py38h497a2fe_1    conda-forge
 pthread-stubs             0.4               h36c2ea0_1001    conda-forge
 ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
 py                        1.10.0                   pypi_0    pypi
 py-lief                   0.11.5           py38h709712a_0    conda-forge
 py-xgboost                1.4.2dev.rapidsai21.10  cuda11.2py38_0    rapidsai
 pyannote-core             4.1                      pypi_0    pypi
 pyannote-database         4.1                      pypi_0    pypi
 pyannote-metrics          3.0.1                    pypi_0    pypi
 pyarrow                   5.0.0           py38hed47224_1_cuda    conda-forge
 pyasn1                    0.4.8                      py_0    conda-forge
 pyasn1-modules            0.2.7                      py_0    conda-forge
 pybind11                  2.6.2                    pypi_0    pypi
 pybtex                    0.24.0                   pypi_0    pypi
 pybtex-docutils           1.0.1                    pypi_0    pypi
 pycocotools               2.0+nv0.5.1              pypi_0    pypi
 pycodestyle               2.5.0                    pypi_0    pypi
 pycosat                   0.6.3           py38h497a2fe_1006    conda-forge
 pycparser                 2.20               pyh9f0ad1d_2    conda-forge
 pyct                      0.4.8                    py38_0
 pycuda                    2020.1                   pypi_0    pypi
 pydantic                  1.7.3            py38h497a2fe_1    conda-forge
 pydeck                    0.5.0              pyh9f0ad1d_0    conda-forge
 pydeprecate               0.3.1                    pypi_0    pypi
 pydot                     1.4.2                    pypi_0    pypi
 pydub                     0.25.1                   pypi_0    pypi
 pyee                      8.1.0              pyh9f0ad1d_0    conda-forge
 pyflakes                  2.1.1                    pypi_0    pypi
 pygments                  2.9.0              pyhd8ed1ab_0    conda-forge
 pynini                    2.1.3            py38h1fd1430_0    conda-forge
 pynvml                    8.0.4                    pypi_0    pypi
 pyopenssl                 20.0.1             pyhd8ed1ab_0    conda-forge
 pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
 pypinyin                  0.42.0                   pypi_0    pypi
 pyppeteer                 0.2.6              pyhd8ed1ab_0    conda-forge
 pyproj                    3.1.0            py38h03a1999_3    conda-forge
 pyrsistent                0.17.3                   pypi_0    pypi
 pysmilesutils             1.0.0                    pypi_0    pypi
 pysocks                   1.7.1            py38h578d9bd_3    conda-forge
 pystoi                    0.3.3                    pypi_0    pypi
 pytablewriter             0.47.0                   pypi_0    pypi
 pytest                    6.2.4                    pypi_0    pypi
 pytest-cov                2.12.1                   pypi_0    pypi
 pytest-pythonpath         0.7.3                    pypi_0    pypi
 pytest-runner             5.3.1                    pypi_0    pypi
 python                    3.8.10          h49503c6_1_cpython    conda-forge
 python-confluent-kafka    1.6.0            py38h497a2fe_1    conda-forge
 python-dateutil           2.8.1                      py_0    conda-forge
 python-dotenv             0.17.1                   pypi_0    pypi
 python-hostlist           1.21                     pypi_0    pypi
 python-libarchive-c       3.1              py38h578d9bd_0    conda-forge
 python-nvd3               0.15.0                   pypi_0    pypi
 python-slugify            5.0.2                    pypi_0    pypi
 python_abi                3.8                      1_cp38    conda-forge
 pytools                   2021.2.7                 pypi_0    pypi
 pytorch-lightning         1.5.1                    pypi_0    pypi
 pytorch-quantization      2.1.0                    pypi_0    pypi
 pytorch-transformers      1.1.0                    pypi_0    pypi
 pytz                      2021.1             pyhd8ed1ab_0    conda-forge
 pyu2f                     0.1.5              pyhd8ed1ab_0    conda-forge
 pyviz_comms               2.0.2              pyhd3eb1b0_0
 pywavelets                1.1.1            py38h7b6447c_2
 pyyaml                    5.4.1            py38h497a2fe_0    conda-forge
 pyzmq                     22.1.0                   pypi_0    pypi
 rapidfuzz                 1.4.1                    pypi_0    pypi
 rapids                    21.10.00        cuda11.2_py38_ge66f011_114    rapidsai
 rapids-xgboost            21.10.00        cuda11.2_py38_ge66f011_114    rapidsai
 rdkit-pypi                2020.9.5.2               pypi_0    pypi
 re2                       2021.06.01           h9c3ff4c_0    conda-forge
 readline                  8.1                  h46c0cb4_0    conda-forge
 regex                     2021.4.4                 pypi_0    pypi
 requests                  2.25.1             pyhd3deb0d_0    conda-forge
 requests-oauthlib         1.3.0                    pypi_0    pypi
 resampy                   0.2.2                    pypi_0    pypi
 revtok                    0.0.3                    pypi_0    pypi
 rhash                     1.4.1                h7f98852_0    conda-forge
 ripgrep                   13.0.0               habb4d0f_0    conda-forge
 rmm                       21.10.01        cuda_11.2_py38_gc54767f_0    rapidsai
 rsa                       4.7.2              pyh44b312d_0    conda-forge
 rtree                     0.9.7            py38h06a4308_1
 ruamel-yaml               0.17.10                  pypi_0    pypi
 ruamel-yaml-clib          0.2.6                    pypi_0    pypi
 ruamel_yaml               0.15.80         py38h497a2fe_1004    conda-forge
 s2n                       1.0.10               h9b69904_0    conda-forge
 s3transfer                0.4.2              pyhd8ed1ab_0    conda-forge
 sacrebleu                 1.5.1                    pypi_0    pypi
 sacremoses                0.0.45                   pypi_0    pypi
 scikit-image              0.15.0                   pypi_0    pypi
 scikit-learn              0.24.2                   pypi_0    pypi
 scipy                     1.6.3            py38h7b17777_0    conda-forge
 send2trash                1.5.0                    pypi_0    pypi
 sentencepiece             0.1.95                   pypi_0    pypi
 sentry-sdk                1.3.1                    pypi_0    pypi
 setuptools                49.6.0           py38h578d9bd_3    conda-forge
 shapely                   1.7.1            py38haeee4fe_5    conda-forge
 shellingham               1.4.0              pyh44b312d_0    conda-forge
 shortuuid                 1.0.1                    pypi_0    pypi
 simpervisor               0.4                pyhd8ed1ab_0    conda-forge
 simplejson                3.17.3                   pypi_0    pypi
 six                       1.16.0             pyh6c4a22f_0    conda-forge
 smart_open                2.2.1              pyh9f0ad1d_0    conda-forge
 smmap                     4.0.0                    pypi_0    pypi
 snappy                    1.1.8                he6710b0_0
 sniffio                   1.2.0            py38h06a4308_1
 snowballstemmer           2.1.0                    pypi_0    pypi
 sortedcontainers          2.4.0              pyhd3eb1b0_0
 soundfile                 0.10.3.post1             pypi_0    pypi
 soupsieve                 2.0.1                      py_1    conda-forge
 sox                       1.4.1                    pypi_0    pypi
 spacy                     3.0.6            py38hfc89cab_1    conda-forge
 spacy-legacy              3.0.6              pyhd8ed1ab_0    conda-forge
 spdlog                    1.9.0                hd09550d_0
 sphinx                    4.0.2                    pypi_0    pypi
 sphinx-glpi-theme         0.3                      pypi_0    pypi
 sphinx-rtd-theme          0.5.2                    pypi_0    pypi
 sphinxcontrib-applehelp   1.0.2                    pypi_0    pypi
 sphinxcontrib-bibtex      2.3.0                    pypi_0    pypi
 sphinxcontrib-devhelp     1.0.2                    pypi_0    pypi
 sphinxcontrib-htmlhelp    2.0.0                    pypi_0    pypi
 sphinxcontrib-jsmath      1.0.1                    pypi_0    pypi
 sphinxcontrib-qthelp      1.0.3                    pypi_0    pypi
 sphinxcontrib-serializinghtml 1.1.5                    pypi_0    pypi
 sqlalchemy                1.3.20                   pypi_0    pypi
 sqlite                    3.35.5               h74cdb3f_0    conda-forge
 sqlparse                  0.4.1                    pypi_0    pypi
 srsly                     2.4.1            py38h709712a_0    conda-forge
 streamz                   0.6.3              pyhd3eb1b0_0
 subword-nmt               0.3.3                    pypi_0    pypi
 sympy                     1.8                      pypi_0    pypi
 tabledata                 1.1.4                    pypi_0    pypi
 tabulate                  0.8.9                    pypi_0    pypi
 tblib                     1.7.0              pyhd3eb1b0_0
 tensorboard               2.4.1                    pypi_0    pypi
 tensorboard-data-server   0.6.1                    pypi_0    pypi
 tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
 tensorboardx              1.8                      pypi_0    pypi
 tensorrt                  7.2.3.4                  pypi_0    pypi
 terminado                 0.10.1                   pypi_0    pypi
 testpath                  0.5.0              pyhd3eb1b0_0
 text-unidecode            1.3                      pypi_0    pypi
 thinc                     8.0.3            py38hfc89cab_1    conda-forge
 threadpoolctl             2.1.0                    pypi_0    pypi
 thrust                    1.11.0               ha770c72_1    conda-forge
 tifffile                  2021.4.8           pyhd3eb1b0_2
 tiledb                    2.3.3                hc47dd91_1
 tk                        8.6.10               h21135ba_1    conda-forge
 tokenizers                0.10.3                   pypi_0    pypi
 toml                      0.10.2                   pypi_0    pypi
 toolz                     0.11.1             pyhd3eb1b0_0
 torch                     1.9.0a0+c3d40fd          pypi_0    pypi
 torch-stft                0.1.4                    pypi_0    pypi
 torchaudio                0.7.0a0+42d447d          pypi_0    pypi
 torchmetrics              0.4.1                    pypi_0    pypi
 torchtext                 0.8.0a0+0f911ec          pypi_0    pypi
 torchvision               0.10.0a0                 pypi_0    pypi
 tornado                   6.1              py38h27cfd23_0
 tqdm                      4.53.0                   pypi_0    pypi
 traitlets                 5.0.5                      py_0    conda-forge
 transformers              4.8.1                    pypi_0    pypi
 treelite                  2.1.0            py38h01cfe54_0    conda-forge
 treelite-runtime          2.1.0                    pypi_0    pypi
 typed-ast                 1.4.3                    pypi_0    pypi
 typepy                    1.1.5                    pypi_0    pypi
 typer                     0.3.2              pyhd8ed1ab_0    conda-forge
 typing-extensions         3.10.0.0             hd8ed1ab_0    conda-forge
 typing_extensions         3.10.0.0           pyha770c72_0    conda-forge
 tzcode                    2021a                h7f98852_2    conda-forge
 tzdata                    2021a                he74cb21_0    conda-forge
 ucx                       1.11.1+gc58db6b      cuda11.2_0    rapidsai
 ucx-proc                  1.0.0                       gpu    rapidsai
 ucx-py                    0.22.0          py38_gc58db6b_0    rapidsai
 uff                       0.6.9                    pypi_0    pypi
 unidecode                 1.2.0                    pypi_0    pypi
 urllib3                   1.26.6                   pypi_0    pypi
 uvicorn                   0.14.0                   pypi_0    pypi
 uvloop                    0.15.2                   pypi_0    pypi
 wandb                     0.11.1                   pypi_0    pypi
 wasabi                    0.8.2              pyh44b312d_0    conda-forge
 watchgod                  0.7                      pypi_0    pypi
 wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
 webdataset                0.1.62                   pypi_0    pypi
 webencodings              0.5.1                    pypi_0    pypi
 websockets                9.1              py38h497a2fe_0    conda-forge
 werkzeug                  2.0.1                    pypi_0    pypi
 wget                      3.2                      pypi_0    pypi
 wheel                     0.36.2             pyhd3deb0d_0    conda-forge
 whitenoise                5.2.0                    pypi_0    pypi
 widgetsnbextension        3.5.1                    py38_0
 wordninja                 2.0.0                    pypi_0    pypi
 wrapt                     1.10.11                  pypi_0    pypi
 xarray                    0.19.0             pyhd3eb1b0_1
 xerces-c                  3.2.3                h9d8b166_2    conda-forge
 xgboost                   1.4.2dev.rapidsai21.10  cuda11.2py38_0    rapidsai
 xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
 xorg-libice               1.0.10               h7f98852_0    conda-forge
 xorg-libsm                1.2.3             hd9c2040_1000    conda-forge
 xorg-libx11               1.7.2                h7f98852_0    conda-forge
 xorg-libxau               1.0.9                h7f98852_0    conda-forge
 xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
 xorg-libxext              1.3.4                h7f98852_1    conda-forge
 xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
 xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
 xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
 xorg-xproto               7.0.31            h7f98852_1007    conda-forge
 xz                        5.2.5                h516909a_1    conda-forge
 yacs                      0.1.8                    pypi_0    pypi
 yaml                      0.2.5                h516909a_0    conda-forge
 yarg                      0.1.9                    pypi_0    pypi
 yarl                      1.5.1            py38h1e0a361_0    conda-forge
 youtokentome              1.0.6                    pypi_0    pypi
 zeromq                    4.3.4                h2531618_0
 zfp                       0.5.5                h2531618_6
 zict                      2.0.0              pyhd3eb1b0_0
 zipp                      3.5.0                    pypi_0    pypi
 zlib                      1.2.11            h36c2ea0_1013    conda-forge
 zstd                      1.5.0                ha95c52a_0    conda-forge

@rilango rilango added Needs Triage Need team to review and classify bug Something isn't working labels Nov 16, 2021
@shwina
Copy link
Contributor

shwina commented Nov 16, 2021

Thank you for reportng @rilango -- is there any chance you could share the contents of the .csv file directly as plain text? Or if not, some representative data that reproduces the error? This will help make debugging much easier for cuDF developers!

@shwina shwina added Python Affects Python cuDF API. and removed Needs Triage Need team to review and classify labels Nov 16, 2021
@rilango
Copy link
Author

rilango commented Nov 16, 2021

@shwina
Copy link
Contributor

shwina commented Nov 16, 2021

Thank you @rilango -- two questions:

  1. I'm unable to access that file with its default sharing settings: should I ask for permission?
  2. Is it alright if I share that data publicly so other developers can help debug as well?

Again, public data is much more preferable here, but I'd understand if the issue is not easily reproducible for a generic dataset.

@rilango
Copy link
Author

rilango commented Nov 16, 2021

@shwina

Sorry. Please try now. It is now shared with public.
https://drive.google.com/file/d/1-hIshmdiYd70BhDnfgYk-BSF89JU3HiG/view?usp=sharing

@shwina
Copy link
Contributor

shwina commented Nov 24, 2021

FYI, I am able to narrow down the repro to:

import cudf

df = cudf.read_csv('ranking.csv', usecols=['1557'])
print(df.rank())

@shwina
Copy link
Contributor

shwina commented Nov 24, 2021

cc: @karthikeyann @davidwendt

@karthikeyann
Copy link
Contributor

repro without csv.

import cudf
df = cudf.DataFrame({'0': range(0, 1557)})
df.rank() #no error
df = cudf.DataFrame({'0': range(0, 1558)})
df.rank() #error

@karthikeyann
Copy link
Contributor

memcheck yield error when size is 1538 (close to ~1024+512)
df = cudf.DataFrame({'0': range(0, 1538)})

It is share memory write out of bounds error in reduce_by_key

========= Invalid __shared__ write of size 4 bytes
=========     at 0x69c0 in _ZN6thrust8cuda_cub4core13_kernel_agentINS0_15__reduce_by_key16ReduceByKeyAgentIPKiNS_18transform_iteratorIZN37_INTERNAL_15441777_7_rank_cu_7da880c14cudf6detail39_GLOBAL__N__15441777_7_rank_cu_7da880c112rank_averageENS9_11device_spanIS5_Lm18446744073709551615EEENS9_11column_viewENS9_19mutable_column_viewEN3rmm16cuda_stream_viewEEUlT_E_NS_17counting_iteratorIiNS_11use_defaultESL_SL_EESL_SL_EENS_16discard_iteratorISL_EEPNS_4pairIiiEENS_8equal_toIiEEZNSB_12rank_averageESD_SE_SF_SH_EUlSI_T0_E_PiiEES6_SN_SP_SS_SX_N3cub24ReduceByKeyScanTileStateISR_iLb0EEESU_SW_iiEEvSV_T1_T2_T3_T4_T5_T6_T7_T8_T9_
=========     by thread (170,0,0) in block (0,0,0)
=========     Address 0x4810 is out of bounds

karthikeyann added a commit that referenced this issue Nov 25, 2021
Fixes #9703
add __host__ for nvcc return type deduction to work. 
replaced `auto` (generic lambda) with size_type.
@karthikeyann karthikeyann self-assigned this Nov 25, 2021
@karthikeyann
Copy link
Contributor

This issue will be fixed by PR #9778

@karthikeyann
Copy link
Contributor

karthikeyann commented Nov 25, 2021

Cause of this issue is that return type is automatically assumed int by nvcc if it can't deduce type of __device__ lambda. (need confirmation)
adding trailing-return-type to the __device__ lambda doesn't work either.
workaround/fix is to add __host__, and remove auto from arguments to allow deduction of return type.
fix is to use device functor with return type specified.

@rapids-bot rapids-bot bot closed this as completed in #9778 Dec 1, 2021
rapids-bot bot pushed a commit that referenced this issue Dec 1, 2021
Fixes #9703
replace device lambda with device functor with return type. (due to [14. extended-lambda-restrictions](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#extended-lambda-restrictions) )
~add `__host__` to lambda for nvcc return type deduction to work properly.~
~replaced `auto` (generic lambda) with `size_type`.~
fixes shared memory write error caused in #9703

Authors:
  - Karthikeyan (https://github.com/karthikeyann)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)
  - David Wendt (https://github.com/davidwendt)
  - Jake Hemstad (https://github.com/jrhemstad)

URL: #9778
@karthikeyann
Copy link
Contributor

Issue is caused by return type deduction of device lambda limitation. Point 14 in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#extended-lambda-restrictions
PR #9778 fixed this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Python Affects Python cuDF API.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants