Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] nvtext::subword_tokenize fails when given only whitespace #13300

Closed
cwharris opened this issue May 5, 2023 · 0 comments · Fixed by #13320
Closed

[BUG] nvtext::subword_tokenize fails when given only whitespace #13300

cwharris opened this issue May 5, 2023 · 0 comments · Fixed by #13320
Assignees
@davidwendt
Labels
bug Something isn't working

Comments

@cwharris
Copy link
Contributor

cwharris commented May 5, 2023 •
edited
Loading

Describe the bug
nvtext_subword_tokenize throws an exception when given only whitespace as input.

Steps/Code to reproduce bug

TEST(TextSubwordTest, WhitespaceOnly)
{
  cudf::test::strings_column_wrapper strings({" "}, {1});
  std::string hash_file = temp_env->get_temp_filepath("hashed_vocab.txt");
  create_hashed_vocab(hash_file);
  auto vocab  = nvtext::load_vocabulary_file(hash_file);
  auto result = nvtext::subword_tokenize(cudf::strings_column_view{strings},
                                         *vocab,
                                         16,
                                         16,
                                         true,   // do_lower_case
                                         true,  // do_truncate
                                         2);
  EXPECT_EQ(uint32_t{0}, result.nrows_tensor);
  EXPECT_EQ(0, result.tensor_token_ids->size());
  EXPECT_EQ(0, result.tensor_attention_mask->size());
  EXPECT_EQ(0, result.tensor_metadata->size());
}
Note: Google Test filter = TextSubwordTest.WhitespaceOnly
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from TextSubwordTest
[ RUN      ] TextSubwordTest.WhitespaceOnly
unknown file: Failure
C++ exception with description "CUDF failure at: /home/charris/dev/rapids/cudf/cpp/include/cudf/detail/utilities/cuda.cuh:66: num_blocks must be > 0" thrown in the test body.
[  FAILED  ] TextSubwordTest.WhitespaceOnly (1 ms)
[----------] 1 test from TextSubwordTest (1 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (1 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] TextSubwordTest.WhitespaceOnly

 1 FAILED TEST

Expected behavior
subword::tokenize should not throw an exception when given only whitespace as input.

Environment overview (please complete the following information)

  • Environment location: Bare-metal
  • Method of cuDF install: from source

Environment details

Click here to see environment details
 **git***
 commit ac60656bc929c08f522d23328a515caa52cee989 (HEAD -> branch-23.02, upstream/branch-23.02)
 Author: Ray Douglass <[email protected]>
 Date:   Thu Feb 9 11:13:21 2023 -0500
 
 Fix update-version.sh (#12745)
 
 Fix the `update-version.sh` script by removing a file that no longer exists
 
 Authors:
 - Ray Douglass (https://github.com/raydouglass)
 
 Approvers:
 - Sevag H (https://github.com/sevagh)
 **git submodules***
 
 ***OS Information***
 DISTRIB_ID=Ubuntu
 DISTRIB_RELEASE=20.04
 DISTRIB_CODENAME=focal
 DISTRIB_DESCRIPTION="Ubuntu 20.04.5 LTS"
 NAME="Ubuntu"
 VERSION="20.04.5 LTS (Focal Fossa)"
 ID=ubuntu
 ID_LIKE=debian
 PRETTY_NAME="Ubuntu 20.04.5 LTS"
 VERSION_ID="20.04"
 HOME_URL="https://www.ubuntu.com/"
 SUPPORT_URL="https://help.ubuntu.com/"
 BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
 PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
 VERSION_CODENAME=focal
 UBUNTU_CODENAME=focal
 Linux charris-dt 5.10.0-1050-oem #52-Ubuntu SMP Wed Oct 6 12:58:49 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
 
 ***GPU Information***
 Fri May  5 11:51:41 2023
 +---------------------------------------------------------------------------------------+
 | NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |
 |-----------------------------------------+----------------------+----------------------+
 | GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
 | Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
 |                                         |                      |               MIG M. |
 |=========================================+======================+======================|
 |   0  Quadro RTX 8000                 On | 00000000:61:00.0 Off |                  Off |
 | 33%   36C    P3               53W / 260W|   2106MiB / 49152MiB |      0%      Default |
 |                                         |                      |                  N/A |
 +-----------------------------------------+----------------------+----------------------+
 
 +---------------------------------------------------------------------------------------+
 | Processes:                                                                            |
 |  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
 |        ID   ID                                                             Usage      |
 |=======================================================================================|
 |    0   N/A  N/A      2200      G   /usr/lib/xorg/Xorg                           77MiB |
 |    0   N/A  N/A      3313      G   /usr/lib/xorg/Xorg                          750MiB |
 |    0   N/A  N/A      3454      G   /usr/bin/gnome-shell                        238MiB |
 |    0   N/A  N/A      4741      G   ...sion,SpareRendererForSitePerProcess      814MiB |
 |    0   N/A  N/A     54534      G   ...65400421,2755869001877925308,131072       98MiB |
 |    0   N/A  N/A   1276865      G   ...,WinRetrieveSuggestionsOnlyOnDemand      114MiB |
 +---------------------------------------------------------------------------------------+
 WARNING: infoROM is corrupted at gpu 0000:61:00.0
 
 ***CPU***
 Architecture:                    x86_64
 CPU op-mode(s):                  32-bit, 64-bit
 Byte Order:                      Little Endian
 Address sizes:                   43 bits physical, 48 bits virtual
 CPU(s):                          64
 On-line CPU(s) list:             0-63
 Thread(s) per core:              2
 Core(s) per socket:              32
 Socket(s):                       1
 NUMA node(s):                    1
 Vendor ID:                       AuthenticAMD
 CPU family:                      23
 Model:                           49
 Model name:                      AMD Ryzen Threadripper PRO 3975WX 32-Cores
 Stepping:                        0
 Frequency boost:                 enabled
 CPU MHz:                         2395.243
 CPU max MHz:                     4368.1641
 CPU min MHz:                     2200.0000
 BogoMIPS:                        6986.46
 Virtualization:                  AMD-V
 L1d cache:                       1 MiB
 L1i cache:                       1 MiB
 L2 cache:                        16 MiB
 L3 cache:                        128 MiB
 NUMA node0 CPU(s):               0-63
 Vulnerability Itlb multihit:     Not affected
 Vulnerability L1tf:              Not affected
 Vulnerability Mds:               Not affected
 Vulnerability Meltdown:          Not affected
 Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
 Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
 Vulnerability Spectre v2:        Mitigation; Full AMD retpoline, IBPB conditional, STIBP conditional, RSB filling
 Vulnerability Srbds:             Not affected
 Vulnerability Tsx async abort:   Not affected
 Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate sme ssbd mba sev ibpb stibp vmmcall sev_es fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca
 
 ***CMake***
 /home/charris/miniconda3/envs/cudf/bin/cmake
 cmake version 3.26.3
 
 CMake suite maintained and supported by Kitware (kitware.com/cmake).
 
 ***g++***
 /home/charris/miniconda3/envs/cudf/bin/g++
 g++ (conda-forge gcc 9.5.0-19) 9.5.0
 Copyright (C) 2019 Free Software Foundation, Inc.
 This is free software; see the source for copying conditions.  There is NO
 warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 
 
 ***nvcc***
 /home/charris/miniconda3/envs/cudf/bin/nvcc
 nvcc: NVIDIA (R) Cuda compiler driver
 Copyright (c) 2005-2022 NVIDIA Corporation
 Built on Wed_Sep_21_10:33:58_PDT_2022
 Cuda compilation tools, release 11.8, V11.8.89
 Build cuda_11.8.r11.8/compiler.31833905_0
 
 ***Python***
 /home/charris/miniconda3/envs/cudf/bin/python
 Python 3.10.10
 
 ***Environment Variables***
 PATH                            : /home/charris/.local/bin:/home/charris/.local/bin:/home/charris/miniconda3/envs/cudf/bin:/home/charris/miniconda3/condabin:/home/charris/.nvm/versions/node/v17.0.1/bin:/home/charris/.local/bin:/home/charris/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/cuda/bin:/usr/local/cuda/bin:/usr/local/cuda/bin
 LD_LIBRARY_PATH                 :
 NUMBAPRO_NVVM                   :
 NUMBAPRO_LIBDEVICE              :
 CONDA_PREFIX                    : /home/charris/miniconda3/envs/cudf
 PYTHON_PATH                     :
 
 ***conda packages***
 /home/charris/miniconda3/condabin/conda
 # packages in environment at /home/charris/miniconda3/envs/cudf:
 #
 # Name                    Version                   Build  Channel
 _libgcc_mutex             0.1                 conda_forge    conda-forge
 _openmp_mutex             4.5                       2_gnu    conda-forge
 _sysroot_linux-64_curr_repodata_hack 3                   h69a702a_13    conda-forge
 accessible-pygments       0.0.4              pyhd8ed1ab_0    conda-forge
 aiobotocore               2.5.0              pyhd8ed1ab_0    conda-forge
 aiohttp                   3.8.4           py310h1fa729e_0    conda-forge
 aioitertools              0.11.0             pyhd8ed1ab_0    conda-forge
 aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
 alabaster                 0.7.13             pyhd8ed1ab_0    conda-forge
 anyio                     3.6.2              pyhd8ed1ab_0    conda-forge
 argon2-cffi               21.3.0             pyhd8ed1ab_0    conda-forge
 argon2-cffi-bindings      21.2.0          py310h5764c6d_3    conda-forge
 arrow-cpp                 10.0.1          ha770c72_23_cpu    conda-forge
 asttokens                 2.2.1              pyhd8ed1ab_0    conda-forge
 async-timeout             4.0.2              pyhd8ed1ab_0    conda-forge
 attrs                     23.1.0             pyh71513ae_0    conda-forge
 aws-c-auth                0.6.26               h2c7c9e7_6    conda-forge
 aws-c-cal                 0.5.26               h71eb795_0    conda-forge
 aws-c-common              0.8.17               hd590300_0    conda-forge
 aws-c-compression         0.2.16               h4f47f36_6    conda-forge
 aws-c-event-stream        0.2.20               h69ce273_6    conda-forge
 aws-c-http                0.7.7                h7b8353a_3    conda-forge
 aws-c-io                  0.13.21              hcccde9c_3    conda-forge
 aws-c-mqtt                0.8.6               h3a1964a_15    conda-forge
 aws-c-s3                  0.2.8                h0933b68_4    conda-forge
 aws-c-sdkutils            0.1.9                h4f47f36_1    conda-forge
 aws-checksums             0.1.14               h4f47f36_6    conda-forge
 aws-crt-cpp               0.19.9               h85076f6_5    conda-forge
 aws-sam-translator        1.55.0             pyhd8ed1ab_0    conda-forge
 aws-sdk-cpp               1.10.57             hf40e4db_10    conda-forge
 aws-xray-sdk              2.12.0             pyhd8ed1ab_0    conda-forge
 babel                     2.12.1             pyhd8ed1ab_1    conda-forge
 backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
 backports                 1.0                pyhd8ed1ab_3    conda-forge
 backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
 backports.zoneinfo        0.2.1           py310hff52083_7    conda-forge
 bcrypt                    3.2.2           py310h5764c6d_1    conda-forge
 beautifulsoup4            4.12.2             pyha770c72_0    conda-forge
 binutils                  2.39                 hdd6e379_1    conda-forge
 binutils_impl_linux-64    2.39                 he00db2b_1    conda-forge
 binutils_linux-64         2.39                h5fc0e48_13    conda-forge
 blas                      1.0                         mkl    conda-forge
 bleach                    6.0.0              pyhd8ed1ab_0    conda-forge
 blinker                   1.6.2              pyhd8ed1ab_0    conda-forge
 bokeh                     2.4.3              pyhd8ed1ab_3    conda-forge
 boto3                     1.26.76            pyhd8ed1ab_0    conda-forge
 botocore                  1.29.76            pyhd8ed1ab_0    conda-forge
 brotlipy                  0.7.0           py310h5764c6d_1005    conda-forge
 bzip2                     1.0.8                h7f98852_4    conda-forge
 c-ares                    1.18.1               h7f98852_0    conda-forge
 c-compiler                1.3.0                h7f98852_0    conda-forge
 ca-certificates           2022.12.7            ha878542_0    conda-forge
 cachetools                5.3.0              pyhd8ed1ab_0    conda-forge
 certifi                   2022.12.7          pyhd8ed1ab_0    conda-forge
 cffi                      1.15.1          py310h255011f_3    conda-forge
 cfgv                      3.3.1              pyhd8ed1ab_0    conda-forge
 cfn-lint                  0.75.1             pyhd8ed1ab_0    conda-forge
 charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
 click                     8.1.3           unix_pyhd8ed1ab_2    conda-forge
 cloudpickle               2.2.1              pyhd8ed1ab_0    conda-forge
 cmake                     3.26.3               h077f3f9_0    conda-forge
 colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
 comm                      0.1.3              pyhd8ed1ab_0    conda-forge
 commonmark                0.9.1                      py_0    conda-forge
 coverage                  7.2.5           py310h2372a71_0    conda-forge
 cryptography              40.0.2          py310h34c0648_0    conda-forge
 cubinlinker               0.2.2           py310hf09951c_0    rapidsai
 cuda-python               11.8.1          py310h01a121a_2    conda-forge
 cudatoolkit               11.8.0              h37601d7_11    conda-forge
 cudf                      23.2.0a0+313.gac60656bc9          pypi_0    pypi
 cupy                      11.6.0          py310h9216885_0    conda-forge
 cxx-compiler              1.3.0                h4bd325d_0    conda-forge
 cyrus-sasl                2.1.27               h9033bb2_6    conda-forge
 cython                    0.29.34         py310heca2aa9_0    conda-forge
 cytoolz                   0.12.0          py310h5764c6d_1    conda-forge
 dask                      2023.1.1           pyhd8ed1ab_0    conda-forge
 dask-core                 2023.1.2a230210 py_g17dff0911_22    dask/label/dev
 dask-cuda                 23.02.01        py310_g2c50668_0    rapidsai
 dask-cudf                 23.2.0a0+313.gac60656bc9.dirty          pypi_0    pypi
 dataclasses               0.8                pyhc8e2a94_3    conda-forge
 datasets                  2.12.0             pyhd8ed1ab_0    conda-forge
 debugpy                   1.6.7           py310heca2aa9_0    conda-forge
 decopatch                 1.4.10             pyhd8ed1ab_0    conda-forge
 decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
 defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
 dill                      0.3.6              pyhd8ed1ab_1    conda-forge
 distlib                   0.3.6              pyhd8ed1ab_0    conda-forge
 distributed               2023.1.1           pyhd8ed1ab_0    conda-forge
 distro                    1.8.0              pyhd8ed1ab_0    conda-forge
 dlpack                    0.5                  h9c3ff4c_0    conda-forge
 docker-py                 6.0.0              pyhd8ed1ab_0    conda-forge
 docutils                  0.19            py310hff52083_1    conda-forge
 doxygen                   1.8.20               had0d8f1_0    conda-forge
 ecdsa                     0.18.0             pyhd8ed1ab_1    conda-forge
 entrypoints               0.4                pyhd8ed1ab_0    conda-forge
 exceptiongroup            1.1.1              pyhd8ed1ab_0    conda-forge
 execnet                   1.9.0              pyhd8ed1ab_0    conda-forge
 executing                 1.2.0              pyhd8ed1ab_0    conda-forge
 expat                     2.5.0                hcb278e6_1    conda-forge
 fastavro                  1.7.4           py310h2372a71_0    conda-forge
 fastrlock                 0.8             py310hd8f1fbe_3    conda-forge
 filelock                  3.12.0             pyhd8ed1ab_0    conda-forge
 flask                     2.3.2              pyhd8ed1ab_0    conda-forge
 flask_cors                3.0.10             pyhd3deb0d_0    conda-forge
 flit-core                 3.8.0              pyhd8ed1ab_0    conda-forge
 freetype                  2.12.1               hca18f0e_1    conda-forge
 frozenlist                1.3.3           py310h5764c6d_0    conda-forge
 fsspec                    2023.4.0           pyh1a96a4e_0    conda-forge
 future                    0.18.3             pyhd8ed1ab_0    conda-forge
 gcc                       9.5.0               h1fea6ba_13    conda-forge
 gcc_impl_linux-64         9.5.0               h99780fb_19    conda-forge
 gcc_linux-64              9.5.0               h4258300_13    conda-forge
 gettext                   0.21.1               h27087fc_0    conda-forge
 gflags                    2.2.2             he1b5a44_1004    conda-forge
 glog                      0.6.0                h6f12383_0    conda-forge
 gmp                       6.2.1                h58526e2_0    conda-forge
 gmpy2                     2.1.2           py310h3ec546c_1    conda-forge
 graphql-core              3.2.3              pyhd8ed1ab_0    conda-forge
 greenlet                  2.0.2           py310heca2aa9_0    conda-forge
 gxx                       9.5.0               h1fea6ba_13    conda-forge
 gxx_impl_linux-64         9.5.0               h99780fb_19    conda-forge
 gxx_linux-64              9.5.0               h43f449f_13    conda-forge
 huggingface_hub           0.14.1             pyhd8ed1ab_0    conda-forge
 hypothesis                6.75.2             pyha770c72_0    conda-forge
 identify                  2.5.24             pyhd8ed1ab_0    conda-forge
 idna                      3.4                pyhd8ed1ab_0    conda-forge
 imagesize                 1.4.1              pyhd8ed1ab_0    conda-forge
 importlib-metadata        6.6.0              pyha770c72_0    conda-forge
 importlib_metadata        6.6.0                hd8ed1ab_0    conda-forge
 iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
 intel-openmp              2022.1.0          h9e868ea_3769
 ipykernel                 6.22.0             pyh210e3f2_0    conda-forge
 ipython                   8.13.2             pyh41d4057_0    conda-forge
 ipython_genutils          0.2.0                      py_1    conda-forge
 itsdangerous              2.1.2              pyhd8ed1ab_0    conda-forge
 jedi                      0.18.2             pyhd8ed1ab_0    conda-forge
 jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
 jmespath                  1.0.1              pyhd8ed1ab_0    conda-forge
 joblib                    1.2.0              pyhd8ed1ab_0    conda-forge
 jschema-to-python         1.2.3              pyhd8ed1ab_0    conda-forge
 jsondiff                  2.0.0              pyhd8ed1ab_0    conda-forge
 jsonpatch                 1.32               pyhd8ed1ab_0    conda-forge
 jsonpickle                2.2.0              pyhd8ed1ab_0    conda-forge
 jsonpointer               2.0                        py_0    conda-forge
 jsonschema                3.2.0              pyhd8ed1ab_3    conda-forge
 junit-xml                 1.9                pyh9f0ad1d_0    conda-forge
 jupyter-cache             0.6.1              pyhd8ed1ab_0    conda-forge
 jupyter_client            8.2.0              pyhd8ed1ab_0    conda-forge
 jupyter_core              5.3.0           py310hff52083_0    conda-forge
 jupyter_events            0.6.3              pyhd8ed1ab_0    conda-forge
 jupyter_server            2.5.0              pyhd8ed1ab_0    conda-forge
 jupyter_server_terminals  0.4.4              pyhd8ed1ab_1    conda-forge
 jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
 kernel-headers_linux-64   3.10.0              h4a8ded7_13    conda-forge
 keyutils                  1.6.1                h166bdaf_0    conda-forge
 krb5                      1.20.1               h81ceb04_0    conda-forge
 lcms2                     2.15                 haa2dc70_1    conda-forge
 ld_impl_linux-64          2.39                 hcc3a1bd_1    conda-forge
 lerc                      4.0.0                h27087fc_0    conda-forge
 libabseil                 20230125.0      cxx17_hcb278e6_1    conda-forge
 libarrow                  10.0.1          h9a2e805_23_cpu    conda-forge
 libblas                   3.9.0            16_linux64_mkl    conda-forge
 libbrotlicommon           1.0.9                h166bdaf_8    conda-forge
 libbrotlidec              1.0.9                h166bdaf_8    conda-forge
 libbrotlienc              1.0.9                h166bdaf_8    conda-forge
 libcblas                  3.9.0            16_linux64_mkl    conda-forge
 libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
 libcurl                   8.0.1                h588be90_0    conda-forge
 libdeflate                1.18                 h0b41bf4_0    conda-forge
 libedit                   3.1.20191231         he28a2e2_2    conda-forge
 libev                     4.33                 h516909a_1    conda-forge
 libevent                  2.1.12               h3358134_0    conda-forge
 libexpat                  2.5.0                hcb278e6_1    conda-forge
 libffi                    3.4.2                h7f98852_5    conda-forge
 libgcc-devel_linux-64     9.5.0               h0a57e50_19    conda-forge
 libgcc-ng                 12.2.0              h65d4601_19    conda-forge
 libgcrypt                 1.10.1               h166bdaf_0    conda-forge
 libgfortran-ng            12.2.0              h69a702a_19    conda-forge
 libgfortran5              12.2.0              h337968e_19    conda-forge
 libgomp                   12.2.0              h65d4601_19    conda-forge
 libgoogle-cloud           2.10.0               hac9eb74_0    conda-forge
 libgpg-error              1.46                 h620e276_0    conda-forge
 libgrpc                   1.54.1               hcf146ea_0    conda-forge
 libgsasl                  1.8.0                         2    conda-forge
 libiconv                  1.17                 h166bdaf_0    conda-forge
 libjpeg-turbo             2.1.5.1              h0b41bf4_0    conda-forge
 liblapack                 3.9.0            16_linux64_mkl    conda-forge
 libllvm11                 11.1.0               he0ac6c6_5    conda-forge
 libnghttp2                1.52.0               h61bc06f_0    conda-forge
 libnsl                    2.0.0                h7f98852_0    conda-forge
 libntlm                   1.4               h7f98852_1002    conda-forge
 libnuma                   2.0.16               h0b41bf4_1    conda-forge
 libpng                    1.6.39               h753d276_0    conda-forge
 libprotobuf               3.21.12              h3eb15da_0    conda-forge
 librdkafka                1.7.0                hb1989a6_1    conda-forge
 librmm                    23.02.00        cuda11_g48e8f2a8_0    rapidsai
 libsanitizer              9.5.0               h2f262e1_19    conda-forge
 libsodium                 1.0.18               h36c2ea0_1    conda-forge
 libsqlite                 3.40.0               h753d276_1    conda-forge
 libssh2                   1.10.0               hf14f497_3    conda-forge
 libstdcxx-devel_linux-64  9.5.0               h0a57e50_19    conda-forge
 libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
 libthrift                 0.18.1               h8fd135c_1    conda-forge
 libtiff                   4.5.0                ha587672_6    conda-forge
 libutf8proc               2.8.0                h166bdaf_0    conda-forge
 libuuid                   2.38.1               h0b41bf4_0    conda-forge
 libuv                     1.44.2               h166bdaf_0    conda-forge
 libwebp-base              1.3.0                h0b41bf4_0    conda-forge
 libxcb                    1.13              h7f98852_1004    conda-forge
 libzlib                   1.2.13               h166bdaf_4    conda-forge
 livereload                2.6.3              pyh9f0ad1d_0    conda-forge
 llvmlite                  0.39.1          py310h58363a5_1    conda-forge
 locket                    1.0.0              pyhd8ed1ab_0    conda-forge
 lz4                       4.3.2           py310h0cfdcf0_0    conda-forge
 lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
 makefun                   1.15.1             pyhd8ed1ab_0    conda-forge
 markdown                  3.4.3              pyhd8ed1ab_0    conda-forge
 markdown-it-py            2.2.0              pyhd8ed1ab_0    conda-forge
 markupsafe                2.1.2           py310h1fa729e_0    conda-forge
 matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
 mdit-py-plugins           0.3.5              pyhd8ed1ab_0    conda-forge
 mdurl                     0.1.0              pyhd8ed1ab_0    conda-forge
 mimesis                   9.0.0              pyhd8ed1ab_0    conda-forge
 mistune                   2.0.5              pyhd8ed1ab_0    conda-forge
 mkl                       2022.1.0           hc2b9512_224
 moto                      4.1.8              pyhd8ed1ab_0    conda-forge
 mpc                       1.3.1                hfe3b2da_0    conda-forge
 mpfr                      4.2.0                hb012696_0    conda-forge
 msgpack-python            1.0.5           py310hdf3cbec_0    conda-forge
 multidict                 6.0.4           py310h1fa729e_0    conda-forge
 multiprocess              0.70.14         py310h5764c6d_3    conda-forge
 myst-nb                   0.17.2             pyhd8ed1ab_0    conda-forge
 myst-parser               0.18.1             pyhd8ed1ab_0    conda-forge
 nbclassic                 1.0.0              pyhb4ecaf3_1    conda-forge
 nbclient                  0.7.4              pyhd8ed1ab_0    conda-forge
 nbconvert                 7.3.1              pyhd8ed1ab_0    conda-forge
 nbconvert-core            7.3.1              pyhd8ed1ab_0    conda-forge
 nbconvert-pandoc          7.3.1              pyhd8ed1ab_0    conda-forge
 nbformat                  5.8.0              pyhd8ed1ab_0    conda-forge
 nbsphinx                  0.9.1              pyhd8ed1ab_0    conda-forge
 ncurses                   6.3                  h27087fc_1    conda-forge
 nest-asyncio              1.5.6              pyhd8ed1ab_0    conda-forge
 networkx                  2.8.8              pyhd8ed1ab_0    conda-forge
 ninja                     1.11.1               h924138e_0    conda-forge
 nodeenv                   1.7.0              pyhd8ed1ab_0    conda-forge
 notebook                  6.5.4              pyha770c72_0    conda-forge
 notebook-shim             0.2.3              pyhd8ed1ab_0    conda-forge
 numba                     0.56.4          py310h0e39c9b_1    conda-forge
 numpy                     1.23.5          py310h53a5b5f_0    conda-forge
 numpydoc                  1.5.0              pyhd8ed1ab_0    conda-forge
 nvcc_linux-64             11.8                h41dc85b_22    conda-forge
 nvtx                      0.2.5           py310h1fa729e_0    conda-forge
 openapi-schema-validator  0.2.3              pyhd8ed1ab_0    conda-forge
 openapi-spec-validator    0.4.0              pyhd8ed1ab_1    conda-forge
 openjpeg                  2.5.0                hfec8fc6_2    conda-forge
 openssl                   3.1.0                hd590300_3    conda-forge
 orc                       1.8.3                hfdbbad2_0    conda-forge
 packaging                 23.1               pyhd8ed1ab_0    conda-forge
 pandas                    1.5.3           py310h9b08913_1    conda-forge
 pandoc                    1.19.2                        0    conda-forge
 pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
 paramiko                  3.1.0              pyhd8ed1ab_0    conda-forge
 parquet-cpp               1.5.1                         2    conda-forge
 parso                     0.8.3              pyhd8ed1ab_0    conda-forge
 partd                     1.4.0              pyhd8ed1ab_0    conda-forge
 pbr                       5.11.1             pyhd8ed1ab_0    conda-forge
 pexpect                   4.8.0              pyh1a96a4e_2    conda-forge
 pickleshare               0.7.5                   py_1003    conda-forge
 pillow                    9.5.0           py310h065c6d2_0    conda-forge
 pip                       23.1.2             pyhd8ed1ab_0    conda-forge
 platformdirs              3.5.0              pyhd8ed1ab_0    conda-forge
 pluggy                    1.0.0              pyhd8ed1ab_5    conda-forge
 pooch                     1.7.0              pyha770c72_3    conda-forge
 pre-commit                3.3.1              pyha770c72_0    conda-forge
 prometheus_client         0.16.0             pyhd8ed1ab_0    conda-forge
 prompt-toolkit            3.0.38             pyha770c72_0    conda-forge
 prompt_toolkit            3.0.38               hd8ed1ab_0    conda-forge
 protobuf                  4.21.12         py310heca2aa9_0    conda-forge
 psutil                    5.9.5           py310h1fa729e_0    conda-forge
 pthread-stubs             0.4               h36c2ea0_1001    conda-forge
 ptxcompiler               0.7.0           py310h01a121a_3    conda-forge
 ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
 pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
 py-cpuinfo                9.0.0              pyhd8ed1ab_0    conda-forge
 pyarrow                   10.0.1          py310he6bfd7f_23_cpu    conda-forge
 pyasn1                    0.4.8                      py_0    conda-forge
 pycparser                 2.21               pyhd8ed1ab_0    conda-forge
 pydata-sphinx-theme       0.13.3             pyhd8ed1ab_0    conda-forge
 pygments                  2.15.1             pyhd8ed1ab_0    conda-forge
 pynacl                    1.5.0           py310h5764c6d_2    conda-forge
 pynvml                    11.4.1             pyhd8ed1ab_0    conda-forge
 pyopenssl                 23.1.1             pyhd8ed1ab_0    conda-forge
 pyorc                     0.8.0           py310hd52fb3e_4    conda-forge
 pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
 pyrsistent                0.19.3          py310h1fa729e_0    conda-forge
 pysocks                   1.7.1              pyha2e5f31_6    conda-forge
 pytest                    7.3.1              pyhd8ed1ab_0    conda-forge
 pytest-benchmark          4.0.0              pyhd8ed1ab_0    conda-forge
 pytest-cases              3.6.14             pyhd8ed1ab_0    conda-forge
 pytest-cov                4.0.0              pyhd8ed1ab_0    conda-forge
 pytest-xdist              3.2.1              pyhd8ed1ab_0    conda-forge
 python                    3.10.10         he550d4f_0_cpython    conda-forge
 python-confluent-kafka    1.7.0           py310h6acc77f_2    conda-forge
 python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
 python-fastjsonschema     2.16.3             pyhd8ed1ab_0    conda-forge
 python-jose               3.3.0              pyh6c4a22f_1    conda-forge
 python-json-logger        2.0.7              pyhd8ed1ab_0    conda-forge
 python-snappy             0.6.1           py310hcee4d7c_0    conda-forge
 python-xxhash             3.2.0           py310h1fa729e_0    conda-forge
 python_abi                3.10                    3_cp310    conda-forge
 pytorch                   1.11.0             py3.10_cpu_0    pytorch
 pytorch-mutex             1.0                         cpu    pytorch
 pytz                      2023.3             pyhd8ed1ab_0    conda-forge
 pywin32-on-windows        0.1.0              pyh1179c8e_3    conda-forge
 pyyaml                    6.0             py310h5764c6d_5    conda-forge
 pyzmq                     25.0.2          py310h059b190_0    conda-forge
 re2                       2023.02.02           hcb278e6_0    conda-forge
 readline                  8.2                  h8228510_1    conda-forge
 recommonmark              0.7.1              pyhd8ed1ab_0    conda-forge
 regex                     2023.5.5        py310h2372a71_0    conda-forge
 requests                  2.29.0             pyhd8ed1ab_0    conda-forge
 responses                 0.18.0             pyhd8ed1ab_0    conda-forge
 rfc3339-validator         0.1.4              pyhd8ed1ab_0    conda-forge
 rfc3986-validator         0.1.1              pyh9f0ad1d_0    conda-forge
 rhash                     1.4.3                h166bdaf_0    conda-forge
 rmm                       23.02.00        cuda11_py310_g48e8f2a8_0    rapidsai
 rsa                       4.9                pyhd8ed1ab_0    conda-forge
 s2n                       1.3.42               h3358134_0    conda-forge
 s3fs                      2023.4.0           pyhd8ed1ab_0    conda-forge
 s3transfer                0.6.1              pyhd8ed1ab_0    conda-forge
 sacremoses                0.0.53             pyhd8ed1ab_0    conda-forge
 sarif-om                  1.0.4              pyhd8ed1ab_0    conda-forge
 scikit-build              0.17.3             pyh4af843d_0    conda-forge
 scipy                     1.10.1          py310h8deb116_0    conda-forge
 sed                       4.8                  he412f7d_0    conda-forge
 send2trash                1.8.2              pyh41d4057_0    conda-forge
 setuptools                67.7.2             pyhd8ed1ab_0    conda-forge
 six                       1.16.0             pyh6c4a22f_0    conda-forge
 snappy                    1.1.10               h9fff704_0    conda-forge
 sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
 snowballstemmer           2.2.0              pyhd8ed1ab_0    conda-forge
 sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
 soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
 spdlog                    1.8.5                h4bd325d_1    conda-forge
 sphinx                    5.3.0              pyhd8ed1ab_0    conda-forge
 sphinx-autobuild          2021.3.14          pyhd8ed1ab_0    conda-forge
 sphinx-copybutton         0.5.2              pyhd8ed1ab_0    conda-forge
 sphinx-markdown-tables    0.0.17             pyh6c4a22f_0    conda-forge
 sphinxcontrib-applehelp   1.0.4              pyhd8ed1ab_0    conda-forge
 sphinxcontrib-devhelp     1.0.2                      py_0    conda-forge
 sphinxcontrib-htmlhelp    2.0.1              pyhd8ed1ab_0    conda-forge
 sphinxcontrib-jsmath      1.0.1                      py_0    conda-forge
 sphinxcontrib-qthelp      1.0.3                      py_0    conda-forge
 sphinxcontrib-serializinghtml 1.1.5              pyhd8ed1ab_2    conda-forge
 sphinxcontrib-websupport  1.2.4              pyhd8ed1ab_1    conda-forge
 sqlalchemy                2.0.12          py310h2372a71_0    conda-forge
 sshpubkeys                3.3.1              pyhd8ed1ab_0    conda-forge
 stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
 streamz                   0.6.4              pyh6c4a22f_0    conda-forge
 strings-udf               23.2.0a0+313.gac60656bc9.dirty          pypi_0    pypi
 sysroot_linux-64          2.17                h4a8ded7_13    conda-forge
 tabulate                  0.9.0              pyhd8ed1ab_1    conda-forge
 tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
 terminado                 0.17.1             pyh41d4057_0    conda-forge
 tinycss2                  1.2.1              pyhd8ed1ab_0    conda-forge
 tk                        8.6.12               h27826a3_0    conda-forge
 tokenizers                0.13.1          py310h633acb5_2    conda-forge
 toml                      0.10.2             pyhd8ed1ab_0    conda-forge
 tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
 toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
 tornado                   6.3             py310h1fa729e_0    conda-forge
 tqdm                      4.65.0             pyhd8ed1ab_1    conda-forge
 traitlets                 5.9.0              pyhd8ed1ab_0    conda-forge
 transformers              4.24.0             pyhd8ed1ab_0    conda-forge
 typing-extensions         4.5.0                hd8ed1ab_0    conda-forge
 typing_extensions         4.5.0              pyha770c72_0    conda-forge
 tzdata                    2023c                h71feb2d_0    conda-forge
 ucx                       1.14.0               h8c404fb_2    conda-forge
 ukkonen                   1.0.1           py310hbf28c38_3    conda-forge
 urllib3                   1.26.15            pyhd8ed1ab_0    conda-forge
 virtualenv                20.23.0            pyhd8ed1ab_0    conda-forge
 wcwidth                   0.2.6              pyhd8ed1ab_0    conda-forge
 webencodings              0.5.1                      py_1    conda-forge
 websocket-client          1.5.1              pyhd8ed1ab_0    conda-forge
 werkzeug                  2.3.3              pyhd8ed1ab_0    conda-forge
 wheel                     0.40.0             pyhd8ed1ab_0    conda-forge
 wrapt                     1.15.0          py310h1fa729e_0    conda-forge
 xmltodict                 0.13.0             pyhd8ed1ab_0    conda-forge
 xorg-libxau               1.0.9                h7f98852_0    conda-forge
 xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
 xxhash                    0.8.1                h0b41bf4_0    conda-forge
 xz                        5.2.6                h166bdaf_0    conda-forge
 yaml                      0.2.5                h7f98852_2    conda-forge
 yarl                      1.9.1           py310h2372a71_0    conda-forge
 zeromq                    4.3.4                h9c3ff4c_1    conda-forge
 zict                      3.0.0              pyhd8ed1ab_0    conda-forge
 zipp                      3.15.0             pyhd8ed1ab_0    conda-forge
 zlib                      1.2.13               h166bdaf_4    conda-forge
 zstd                      1.5.2                h3eb15da_6    conda-forge

Additional context
CUDF 23.02
@cwharris cwharris added bug Something isn't working Needs Triage Need team to review and classify labels May 5, 2023
@davidwendt davidwendt mentioned this issue May 9, 2023
Merged
3 tasks
@rapids-bot rapids-bot bot closed this as completed in 403c83f May 15, 2023
@bdice bdice removed the Needs Triage Need team to review and classify label Mar 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@davidwendt davidwendt

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix subword_tokenize error when input contains no tokens davidwendt/cudf
3 participants
@cwharris @bdice @davidwendt