Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] cudaErrorIllegalAddress when trying to write a large list column to json #13030

Closed
galipremsagar opened this issue Mar 29, 2023 · 3 comments · Fixed by #13108
Closed

[BUG] cudaErrorIllegalAddress when trying to write a large list column to json #13030

galipremsagar opened this issue Mar 29, 2023 · 3 comments · Fixed by #13108
Assignees
@karthikeyann
Labels
1 - On Deck To be worked on next bug Something isn't working cuIO cuIO issue libcudf Affects libcudf (C++/CUDA) code.

Comments

@galipremsagar
Copy link
Contributor

Describe the bug
When we try writing a dataframe that has ListColumn with large number of rows, we are getting a runtime error that says cudaErrorIllegalAddress.

Steps/Code to reproduce bug
with small number of rows it works fine:

In [1]: import pandas as pd

In [2]: import cudf

In [3]: import numpy as np

In [4]: size = 10000
   ...: pdf = pd.DataFrame({
   ...:     "lists": [[1, 2], [3], [4, None], None, [1, 2]]*int(size/5),
   ...:     })
   ...: gdf = cudf.from_pandas(pdf)
   ...: gdf.to_json("cudf.json", engine="cudf", orient='records', lines=True)

Right as we increase the number of rows we see the issue:

In [5]: size = 10000000
   ...: pdf = pd.DataFrame({
   ...:     "lists": [[1, 2], [3], [4, None], None, [1, 2]]*int(size/5),
   ...:     })
   ...: gdf = cudf.from_pandas(pdf)
   ...: gdf.to_json("cudf.json", engine="cudf", orient='records', lines=True)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[5], line 8
      2 pdf = pd.DataFrame({
      3     "lists": [[1, 2], [3], [4, None], None, [1, 2]]*int(size/5),
      4     })
      7 gdf = cudf.from_pandas(pdf)
----> 8 gdf.to_json("cudf.json", engine="cudf", orient='records', lines=True)

File /nvme/0/pgali/envs/cudfdev/lib/python3.10/contextlib.py:79, in ContextDecorator.__call__.<locals>.inner(*args, **kwds)
     76 @wraps(func)
     77 def inner(*args, **kwds):
     78     with self._recreate_cm():
---> 79         return func(*args, **kwds)

File /nvme/0/pgali/envs/cudfdev/lib/python3.10/site-packages/cudf/core/frame.py:2548, in Frame.to_json(self, path_or_buf, *args, **kwargs)
   2543 @_cudf_nvtx_annotate
   2544 @ioutils.doc_to_json()
   2545 def to_json(self, path_or_buf=None, *args, **kwargs):
   2546     """{docstring}"""
-> 2548     return cudf.io.json.to_json(
   2549         self, path_or_buf=path_or_buf, *args, **kwargs
   2550     )

File /nvme/0/pgali/envs/cudfdev/lib/python3.10/site-packages/cudf/io/json.py:221, in to_json(cudf_val, path_or_buf, engine, orient, storage_options, *args, **kwargs)
    217         libjson.write_json(
    218             cudf_val, path_or_buf=file_obj, *args, **kwargs
    219         )
    220 else:
--> 221     libjson.write_json(
    222         cudf_val, path_or_buf=path_or_buf, *args, **kwargs
    223     )
    225 if return_as_string:
    226     path_or_buf.seek(0)

File /nvme/0/pgali/envs/cudfdev/lib/python3.10/contextlib.py:79, in ContextDecorator.__call__.<locals>.inner(*args, **kwds)
     76 @wraps(func)
     77 def inner(*args, **kwds):
     78     with self._recreate_cm():
---> 79         return func(*args, **kwds)

File json.pyx:206, in cudf._lib.json.write_json()

RuntimeError: for_each: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered

Somethings to note:

  • This appears to not be an OOM, as I'm able to write a similarly sized string column and This is happening both on T4 & H100.

Expected behavior
Should be able to write the column to dataframe.

Environment overview (please complete the following information)

  • Environment location: [Bare-metal]
  • Method of cuDF install: [from source]

Environment details
Please run and paste the output of the cudf/print_env.sh script here, to gather any other relevant environment details

Click here to see environment details
 **git***
 commit 79fcccfbb4ecea77a9be608a20c0f768063cf2a5 (HEAD -> fix_ci, origin/fix_ci)
 Author: galipremsagar <[email protected]>
 Date:   Tue Mar 28 13:24:27 2023 -0700
 
 Fix versions
 **git submodules***
 
 ***OS Information***
 DISTRIB_ID=Ubuntu
 DISTRIB_RELEASE=18.04
 DISTRIB_CODENAME=bionic
 DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"
 NAME="Ubuntu"
 VERSION="18.04.4 LTS (Bionic Beaver)"
 ID=ubuntu
 ID_LIKE=debian
 PRETTY_NAME="Ubuntu 18.04.4 LTS"
 VERSION_ID="18.04"
 HOME_URL="https://www.ubuntu.com/"
 SUPPORT_URL="https://help.ubuntu.com/"
 BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
 PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
 VERSION_CODENAME=bionic
 UBUNTU_CODENAME=bionic
 Linux dt07 4.15.0-76-generic #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
 
 ***GPU Information***
 Tue Mar 28 19:45:16 2023
 +---------------------------------------------------------------------------------------+
 | NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |
 |-----------------------------------------+----------------------+----------------------+
 | GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
 | Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
 |                                         |                      |               MIG M. |
 |=========================================+======================+======================|
 |   0  Tesla T4                        On | 00000000:3B:00.0 Off |                    0 |
 | N/A   47C    P0               27W /  70W|    496MiB / 15360MiB |      0%      Default |
 |                                         |                      |                  N/A |
 +-----------------------------------------+----------------------+----------------------+
 |   1  Tesla T4                        On | 00000000:5E:00.0 Off |                    0 |
 | N/A   35C    P8                9W /  70W|      4MiB / 15360MiB |      0%      Default |
 |                                         |                      |                  N/A |
 +-----------------------------------------+----------------------+----------------------+
 |   2  Tesla T4                        On | 00000000:AF:00.0 Off |                    0 |
 | N/A   30C    P8               10W /  70W|      4MiB / 15360MiB |      0%      Default |
 |                                         |                      |                  N/A |
 +-----------------------------------------+----------------------+----------------------+
 |   3  Tesla T4                        On | 00000000:D8:00.0 Off |                    0 |
 | N/A   29C    P8                9W /  70W|      4MiB / 15360MiB |      0%      Default |
 |                                         |                      |                  N/A |
 +-----------------------------------------+----------------------+----------------------+
 
 +---------------------------------------------------------------------------------------+
 | Processes:                                                                            |
 |  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
 |        ID   ID                                                             Usage      |
 |=======================================================================================|
 |    0   N/A  N/A     17347      C   ...0/pgali/envs/cudfdev/bin/python3.10      492MiB |
 +---------------------------------------------------------------------------------------+
 
 ***CPU***
 Architecture:        x86_64
 CPU op-mode(s):      32-bit, 64-bit
 Byte Order:          Little Endian
 CPU(s):              64
 On-line CPU(s) list: 0-63
 Thread(s) per core:  2
 Core(s) per socket:  16
 Socket(s):           2
 NUMA node(s):        2
 Vendor ID:           GenuineIntel
 CPU family:          6
 Model:               85
 Model name:          Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
 Stepping:            4
 CPU MHz:             1823.788
 BogoMIPS:            4200.00
 Virtualization:      VT-x
 L1d cache:           32K
 L1i cache:           32K
 L2 cache:            1024K
 L3 cache:            22528K
 NUMA node0 CPU(s):   0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62
 NUMA node1 CPU(s):   1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63
 Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke md_clear flush_l1d
 
 ***CMake***
 /nvme/0/pgali/envs/cudfdev/bin/cmake
 cmake version 3.26.1
 
 CMake suite maintained and supported by Kitware (kitware.com/cmake).
 
 ***g++***
 /nvme/0/pgali/envs/cudfdev/bin/g++
 g++ (conda-forge gcc 11.3.0-19) 11.3.0
 Copyright (C) 2021 Free Software Foundation, Inc.
 This is free software; see the source for copying conditions.  There is NO
 warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 
 
 ***nvcc***
 /nvme/0/pgali/envs/cudfdev/bin/nvcc
 nvcc: NVIDIA (R) Cuda compiler driver
 Copyright (c) 2005-2022 NVIDIA Corporation
 Built on Wed_Sep_21_10:33:58_PDT_2022
 Cuda compilation tools, release 11.8, V11.8.89
 Build cuda_11.8.r11.8/compiler.31833905_0
 
 ***Python***
 /nvme/0/pgali/envs/cudfdev/bin/python
 Python 3.10.10
 
 ***Environment Variables***
 PATH                            : /nvme/0/pgali/envs/cudfdev/bin:/nvme/0/pgali/envs/cudfdev/bin:/nvme/0/pgali/anaconda3/bin:/nvme/0/pgali/envs/cudfdev/bin:/nvme/0/pgali/.cargo/bin:/home/nfs/pgali/.vscode-server/bin/ee2b180d582a7f601fa6ecfdad8d9fd269ab1884/bin/remote-cli:/nvme/0/pgali/.cargo/bin:/nvme/0/pgali/anaconda3/bin:/nvme/0/pgali/anaconda3/condabin:/nvme/0/pgali/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/cuda/bin:/usr/local/cuda/bin
 LD_LIBRARY_PATH                 : /usr/local/cuda/lib64:/usr/local/cuda/lib64::/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64
 NUMBAPRO_NVVM                   :
 NUMBAPRO_LIBDEVICE              :
 CONDA_PREFIX                    : /nvme/0/pgali/envs/cudfdev
 PYTHON_PATH                     :
 
 ***conda packages***
 /nvme/0/pgali/anaconda3/bin/conda
 # packages in environment at /nvme/0/pgali/envs/cudfdev:
 #
 # Name                    Version                   Build  Channel
 _libgcc_mutex             0.1                 conda_forge    conda-forge
 _openmp_mutex             4.5                       2_gnu    conda-forge
 _sysroot_linux-64_curr_repodata_hack 3                   h5bd9786_13    conda-forge
 accessible-pygments       0.0.4              pyhd8ed1ab_0    conda-forge
 aiobotocore               2.4.2              pyhd8ed1ab_0    conda-forge
 aiohttp                   3.8.4           py310h1fa729e_0    conda-forge
 aioitertools              0.11.0             pyhd8ed1ab_0    conda-forge
 aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
 alabaster                 0.7.13             pyhd8ed1ab_0    conda-forge
 anyio                     3.6.2              pyhd8ed1ab_0    conda-forge
 argon2-cffi               21.3.0             pyhd8ed1ab_0    conda-forge
 argon2-cffi-bindings      21.2.0          py310h5764c6d_3    conda-forge
 arrow-cpp                 11.0.0          ha770c72_13_cpu    conda-forge
 asttokens                 2.2.1              pyhd8ed1ab_0    conda-forge
 async-timeout             4.0.2              pyhd8ed1ab_0    conda-forge
 attrs                     22.2.0             pyh71513ae_0    conda-forge
 aws-c-auth                0.6.26               hf365957_1    conda-forge
 aws-c-cal                 0.5.21               h48707d8_2    conda-forge
 aws-c-common              0.8.14               h0b41bf4_0    conda-forge
 aws-c-compression         0.2.16               h03acc5a_5    conda-forge
 aws-c-event-stream        0.2.20               h00877a2_4    conda-forge
 aws-c-http                0.7.6                hf342b9f_0    conda-forge
 aws-c-io                  0.13.19              h5b20300_3    conda-forge
 aws-c-mqtt                0.8.6               hc4349f7_12    conda-forge
 aws-c-s3                  0.2.7                h909e904_1    conda-forge
 aws-c-sdkutils            0.1.8                h03acc5a_0    conda-forge
 aws-checksums             0.1.14               h03acc5a_5    conda-forge
 aws-crt-cpp               0.19.8              hf7fbfca_12    conda-forge
 aws-sam-translator        1.55.0             pyhd8ed1ab_0    conda-forge
 aws-sdk-cpp               1.10.57              h17c43bd_8    conda-forge
 aws-xray-sdk              2.11.0             pyhd8ed1ab_0    conda-forge
 babel                     2.12.1             pyhd8ed1ab_1    conda-forge
 backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
 backports                 1.0                pyhd8ed1ab_3    conda-forge
 backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
 backports.zoneinfo        0.2.1           py310hff52083_7    conda-forge
 bcrypt                    3.2.2           py310h5764c6d_1    conda-forge
 beautifulsoup4            4.12.0             pyha770c72_0    conda-forge
 binutils                  2.39                 hdd6e379_1    conda-forge
 binutils_impl_linux-64    2.39                 he00db2b_1    conda-forge
 binutils_linux-64         2.39                h5fc0e48_12    conda-forge
 blas                      1.0                         mkl    conda-forge
 bleach                    6.0.0              pyhd8ed1ab_0    conda-forge
 bokeh                     2.4.3              pyhd8ed1ab_3    conda-forge
 boto3                     1.24.59            pyhd8ed1ab_0    conda-forge
 botocore                  1.27.59            pyhd8ed1ab_0    conda-forge
 brotlipy                  0.7.0           py310h5764c6d_1005    conda-forge
 bzip2                     1.0.8                h7f98852_4    conda-forge
 c-ares                    1.18.1               h7f98852_0    conda-forge
 c-compiler                1.5.2                h0b41bf4_0    conda-forge
 ca-certificates           2022.12.7            ha878542_0    conda-forge
 cachetools                5.3.0              pyhd8ed1ab_0    conda-forge
 certifi                   2022.12.7          pyhd8ed1ab_0    conda-forge
 cffi                      1.15.1          py310h255011f_3    conda-forge
 cfgv                      3.3.1              pyhd8ed1ab_0    conda-forge
 cfn-lint                  0.75.1             pyhd8ed1ab_0    conda-forge
 charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
 click                     8.1.3           unix_pyhd8ed1ab_2    conda-forge
 cloudpickle               2.2.1              pyhd8ed1ab_0    conda-forge
 cmake                     3.26.1               h077f3f9_0    conda-forge
 colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
 comm                      0.1.3              pyhd8ed1ab_0    conda-forge
 commonmark                0.9.1                      py_0    conda-forge
 coverage                  7.2.2           py310h1fa729e_0    conda-forge
 cryptography              40.0.1          py310h34c0648_0    conda-forge
 cubinlinker               0.2.2           py310hf09951c_0    rapidsai
 cuda-python               11.8.1          py310h01a121a_2    conda-forge
 cuda-sanitizer-api        11.8.86                       0    nvidia
 cudatoolkit               11.8.0              h37601d7_11    conda-forge
 cudf                      23.6.0                   pypi_0    pypi
 cupy                      11.6.0          py310h9216885_0    conda-forge
 cxx-compiler              1.5.2                hf52228f_0    conda-forge
 cyrus-sasl                2.1.27               h9033bb2_6    conda-forge
 cython                    0.29.33         py310heca2aa9_0    conda-forge
 cytoolz                   0.12.0          py310h5764c6d_1    conda-forge
 dask                      2023.3.3a230328  py_gab95634a_6    dask/label/dev
 dask-core                 2023.3.3a230328 py_g55dfbb0ee_6    dask/label/dev
 dask-cuda                 23.06.00a       py310_230328_g7b72914_2    rapidsai-nightly
 dask-cudf                 23.6.0                   pypi_0    pypi
 dataclasses               0.8                pyhc8e2a94_3    conda-forge
 datasets                  1.18.3             pyhd8ed1ab_0    conda-forge
 debugpy                   1.6.6           py310heca2aa9_0    conda-forge
 decopatch                 1.4.10             pyhd8ed1ab_0    conda-forge
 decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
 defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
 dill                      0.3.6              pyhd8ed1ab_1    conda-forge
 distlib                   0.3.6              pyhd8ed1ab_0    conda-forge
 distributed               2023.3.3a230328  py_gab95634a_6    dask/label/dev
 distro                    1.8.0              pyhd8ed1ab_0    conda-forge
 dlpack                    0.5                  h9c3ff4c_0    conda-forge
 docker-py                 6.0.0              pyhd8ed1ab_0    conda-forge
 docutils                  0.19            py310hff52083_1    conda-forge
 doxygen                   1.8.20               had0d8f1_0    conda-forge
 ecdsa                     0.18.0             pyhd8ed1ab_1    conda-forge
 entrypoints               0.4                pyhd8ed1ab_0    conda-forge
 exceptiongroup            1.1.1              pyhd8ed1ab_0    conda-forge
 execnet                   1.9.0              pyhd8ed1ab_0    conda-forge
 executing                 1.2.0              pyhd8ed1ab_0    conda-forge
 expat                     2.5.0                h27087fc_0    conda-forge
 fastavro                  1.7.3           py310h1fa729e_0    conda-forge
 fastrlock                 0.8             py310hd8f1fbe_3    conda-forge
 filelock                  3.10.7             pyhd8ed1ab_0    conda-forge
 flask                     2.1.3              pyhd8ed1ab_0    conda-forge
 flask_cors                3.0.10             pyhd3deb0d_0    conda-forge
 flit-core                 3.8.0              pyhd8ed1ab_0    conda-forge
 fmt                       9.1.0                h924138e_0    conda-forge
 freetype                  2.12.1               hca18f0e_1    conda-forge
 frozenlist                1.3.3           py310h5764c6d_0    conda-forge
 fsspec                    2023.3.0           pyhd8ed1ab_1    conda-forge
 future                    0.18.3             pyhd8ed1ab_0    conda-forge
 gcc                       11.3.0              h02d0930_12    conda-forge
 gcc_impl_linux-64         11.3.0              hab1b70f_19    conda-forge
 gcc_linux-64              11.3.0              he6f903b_12    conda-forge
 gflags                    2.2.2             he1b5a44_1004    conda-forge
 glog                      0.6.0                h6f12383_0    conda-forge
 gmock                     1.10.0               h4bd325d_7    conda-forge
 gmp                       6.2.1                h58526e2_0    conda-forge
 gmpy2                     2.1.2           py310h3ec546c_1    conda-forge
 graphql-core              3.2.3              pyhd8ed1ab_0    conda-forge
 greenlet                  2.0.2           py310heca2aa9_0    conda-forge
 gtest                     1.10.0               h4bd325d_7    conda-forge
 gxx                       11.3.0              h02d0930_12    conda-forge
 gxx_impl_linux-64         11.3.0              hab1b70f_19    conda-forge
 gxx_linux-64              11.3.0              hc203a17_12    conda-forge
 heapdict                  1.0.1                      py_0    conda-forge
 huggingface_hub           0.13.3             pyhd8ed1ab_0    conda-forge
 hypothesis                6.70.1             pyha770c72_0    conda-forge
 identify                  2.5.22             pyhd8ed1ab_0    conda-forge
 idna                      3.4                pyhd8ed1ab_0    conda-forge
 imagesize                 1.4.1              pyhd8ed1ab_0    conda-forge
 importlib-metadata        6.1.0              pyha770c72_0    conda-forge
 importlib_metadata        6.1.0                hd8ed1ab_0    conda-forge
 iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
 intel-openmp              2022.1.0          h9e868ea_3769
 ipykernel                 6.22.0             pyh210e3f2_0    conda-forge
 ipython                   8.11.0             pyh41d4057_0    conda-forge
 ipython_genutils          0.2.0                      py_1    conda-forge
 itsdangerous              2.1.2              pyhd8ed1ab_0    conda-forge
 jedi                      0.18.2             pyhd8ed1ab_0    conda-forge
 jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
 jmespath                  1.0.1              pyhd8ed1ab_0    conda-forge
 joblib                    1.2.0              pyhd8ed1ab_0    conda-forge
 jschema-to-python         1.2.3              pyhd8ed1ab_0    conda-forge
 jsondiff                  2.0.0              pyhd8ed1ab_0    conda-forge
 jsonpatch                 1.32               pyhd8ed1ab_0    conda-forge
 jsonpickle                2.2.0              pyhd8ed1ab_0    conda-forge
 jsonpointer               2.0                        py_0    conda-forge
 jsonschema                3.2.0              pyhd8ed1ab_3    conda-forge
 junit-xml                 1.9                pyh9f0ad1d_0    conda-forge
 jupyter-cache             0.5.0              pyhd8ed1ab_0    conda-forge
 jupyter_client            8.1.0              pyhd8ed1ab_0    conda-forge
 jupyter_core              5.3.0           py310hff52083_0    conda-forge
 jupyter_events            0.6.3              pyhd8ed1ab_0    conda-forge
 jupyter_server            2.5.0              pyhd8ed1ab_0    conda-forge
 jupyter_server_terminals  0.4.4              pyhd8ed1ab_1    conda-forge
 jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
 kernel-headers_linux-64   3.10.0              h4a8ded7_13    conda-forge
 keyutils                  1.6.1                h166bdaf_0    conda-forge
 krb5                      1.20.1               h81ceb04_0    conda-forge
 lcms2                     2.15                 haa2dc70_1    conda-forge
 ld_impl_linux-64          2.39                 hcc3a1bd_1    conda-forge
 lerc                      4.0.0                h27087fc_0    conda-forge
 libabseil                 20230125.0      cxx17_hcb278e6_1    conda-forge
 libarrow                  11.0.0          h93537a5_13_cpu    conda-forge
 libblas                   3.9.0            16_linux64_mkl    conda-forge
 libbrotlicommon           1.0.9                h166bdaf_8    conda-forge
 libbrotlidec              1.0.9                h166bdaf_8    conda-forge
 libbrotlienc              1.0.9                h166bdaf_8    conda-forge
 libcblas                  3.9.0            16_linux64_mkl    conda-forge
 libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
 libcurl                   7.88.1               hdc1c0ab_1    conda-forge
 libdeflate                1.18                 h0b41bf4_0    conda-forge
 libedit                   3.1.20191231         he28a2e2_2    conda-forge
 libev                     4.33                 h516909a_1    conda-forge
 libevent                  2.1.10               h28343ad_4    conda-forge
 libffi                    3.4.2                h7f98852_5    conda-forge
 libgcc-devel_linux-64     11.3.0              h210ce93_19    conda-forge
 libgcc-ng                 12.2.0              h65d4601_19    conda-forge
 libgfortran-ng            12.2.0              h69a702a_19    conda-forge
 libgfortran5              12.2.0              h337968e_19    conda-forge
 libgomp                   12.2.0              h65d4601_19    conda-forge
 libgoogle-cloud           2.8.0                h0bc5f78_1    conda-forge
 libgrpc                   1.52.1               hcf146ea_1    conda-forge
 libiconv                  1.17                 h166bdaf_0    conda-forge
 libjpeg-turbo             2.1.5.1              h0b41bf4_0    conda-forge
 liblapack                 3.9.0            16_linux64_mkl    conda-forge
 libllvm11                 11.1.0               he0ac6c6_5    conda-forge
 libnghttp2                1.52.0               h61bc06f_0    conda-forge
 libnsl                    2.0.0                h7f98852_0    conda-forge
 libntlm                   1.4               h7f98852_1002    conda-forge
 libnuma                   2.0.16               h0b41bf4_1    conda-forge
 libpng                    1.6.39               h753d276_0    conda-forge
 libprotobuf               3.21.12              h3eb15da_0    conda-forge
 librdkafka                1.9.2                ha5a0de0_2    conda-forge
 librmm                    23.06.00a       cuda11_230323_gbc9155e9_0    rapidsai-nightly
 libsanitizer              11.3.0              h239ccf8_19    conda-forge
 libsodium                 1.0.18               h36c2ea0_1    conda-forge
 libsqlite                 3.40.0               h753d276_0    conda-forge
 libssh2                   1.10.0               hf14f497_3    conda-forge
 libstdcxx-devel_linux-64  11.3.0              h210ce93_19    conda-forge
 libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
 libthrift                 0.18.1               h5e4af38_0    conda-forge
 libtiff                   4.5.0                ha587672_6    conda-forge
 libutf8proc               2.8.0                h166bdaf_0    conda-forge
 libuuid                   2.32.1            h7f98852_1000    conda-forge
 libuv                     1.44.2               h166bdaf_0    conda-forge
 libwebp-base              1.3.0                h0b41bf4_0    conda-forge
 libxcb                    1.13              h7f98852_1004    conda-forge
 libzlib                   1.2.13               h166bdaf_4    conda-forge
 livereload                2.6.3              pyh9f0ad1d_0    conda-forge
 llvmlite                  0.39.1          py310h58363a5_1    conda-forge
 locket                    1.0.0              pyhd8ed1ab_0    conda-forge
 lz4                       4.3.2           py310h0cfdcf0_0    conda-forge
 lz4-c                     1.9.4                hcb278e6_0    conda-forge
 make                      4.3                  hd18ef5c_1    conda-forge
 makefun                   1.15.1             pyhd8ed1ab_0    conda-forge
 markdown                  3.4.3              pyhd8ed1ab_0    conda-forge
 markdown-it-py            2.2.0              pyhd8ed1ab_0    conda-forge
 markupsafe                2.1.2           py310h1fa729e_0    conda-forge
 matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
 mdit-py-plugins           0.3.5              pyhd8ed1ab_0    conda-forge
 mdurl                     0.1.0              pyhd8ed1ab_0    conda-forge
 mimesis                   7.0.0              pyhd8ed1ab_0    conda-forge
 mistune                   2.0.5              pyhd8ed1ab_0    conda-forge
 mkl                       2022.1.0           hc2b9512_224
 moto                      4.1.6              pyhd8ed1ab_0    conda-forge
 mpc                       1.3.1                hfe3b2da_0    conda-forge
 mpfr                      4.2.0                hb012696_0    conda-forge
 msgpack-python            1.0.5           py310hdf3cbec_0    conda-forge
 multidict                 6.0.4           py310h1fa729e_0    conda-forge
 multiprocess              0.70.14         py310h5764c6d_3    conda-forge
 myst-nb                   0.17.1             pyhd8ed1ab_0    conda-forge
 myst-parser               0.18.1             pyhd8ed1ab_0    conda-forge
 nbclassic                 0.5.3              pyhb4ecaf3_3    conda-forge
 nbclient                  0.5.13             pyhd8ed1ab_0    conda-forge
 nbconvert                 7.2.9              pyhd8ed1ab_0    conda-forge
 nbconvert-core            7.2.9              pyhd8ed1ab_0    conda-forge
 nbconvert-pandoc          7.2.9              pyhd8ed1ab_0    conda-forge
 nbformat                  5.8.0              pyhd8ed1ab_0    conda-forge
 nbsphinx                  0.9.1              pyhd8ed1ab_0    conda-forge
 ncurses                   6.3                  h27087fc_1    conda-forge
 nest-asyncio              1.5.6              pyhd8ed1ab_0    conda-forge
 networkx                  2.8.8              pyhd8ed1ab_0    conda-forge
 ninja                     1.11.1               h924138e_0    conda-forge
 nodeenv                   1.7.0              pyhd8ed1ab_0    conda-forge
 notebook                  6.5.3              pyha770c72_0    conda-forge
 notebook-shim             0.2.2              pyhd8ed1ab_0    conda-forge
 numba                     0.56.4          py310ha5257ce_0    conda-forge
 numpy                     1.23.5          py310h53a5b5f_0    conda-forge
 numpydoc                  1.5.0              pyhd8ed1ab_0    conda-forge
 nvcc_linux-64             11.8                ha67cc55_22    conda-forge
 nvtx                      0.2.3           py310h5764c6d_2    conda-forge
 openapi-schema-validator  0.2.3              pyhd8ed1ab_0    conda-forge
 openapi-spec-validator    0.4.0              pyhd8ed1ab_1    conda-forge
 openjpeg                  2.5.0                hfec8fc6_2    conda-forge
 openssl                   3.1.0                h0b41bf4_0    conda-forge
 orc                       1.8.3                hfdbbad2_0    conda-forge
 packaging                 23.0               pyhd8ed1ab_0    conda-forge
 pandas                    1.5.3           py310h9b08913_0    conda-forge
 pandoc                    3.1.1                h32600fe_0    conda-forge
 pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
 paramiko                  3.1.0              pyhd8ed1ab_0    conda-forge
 parquet-cpp               1.5.1                         2    conda-forge
 parso                     0.8.3              pyhd8ed1ab_0    conda-forge
 partd                     1.3.0              pyhd8ed1ab_0    conda-forge
 pbr                       5.11.1             pyhd8ed1ab_0    conda-forge
 pexpect                   4.8.0              pyh1a96a4e_2    conda-forge
 pickleshare               0.7.5                   py_1003    conda-forge
 pillow                    9.4.0           py310h065c6d2_2    conda-forge
 pip                       23.0.1             pyhd8ed1ab_0    conda-forge
 platformdirs              3.2.0              pyhd8ed1ab_0    conda-forge
 pluggy                    1.0.0              pyhd8ed1ab_5    conda-forge
 pooch                     1.7.0              pyha770c72_3    conda-forge
 pre-commit                3.2.0              pyha770c72_0    conda-forge
 prometheus_client         0.16.0             pyhd8ed1ab_0    conda-forge
 prompt-toolkit            3.0.38             pyha770c72_0    conda-forge
 prompt_toolkit            3.0.38               hd8ed1ab_0    conda-forge
 protobuf                  4.21.12         py310heca2aa9_0    conda-forge
 psutil                    5.9.4           py310h5764c6d_0    conda-forge
 pthread-stubs             0.4               h36c2ea0_1001    conda-forge
 ptxcompiler               0.7.0           py310h01a121a_3    conda-forge
 ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
 pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
 py-cpuinfo                9.0.0              pyhd8ed1ab_0    conda-forge
 pyarrow                   11.0.0          py310h633f555_13_cpu    conda-forge
 pyasn1                    0.4.8                      py_0    conda-forge
 pycparser                 2.21               pyhd8ed1ab_0    conda-forge
 pydata-sphinx-theme       0.13.1             pyhd8ed1ab_0    conda-forge
 pygments                  2.14.0             pyhd8ed1ab_0    conda-forge
 pynacl                    1.5.0           py310h5764c6d_2    conda-forge
 pynvml                    11.4.1             pyhd8ed1ab_0    conda-forge
 pyopenssl                 23.1.0             pyhd8ed1ab_0    conda-forge
 pyorc                     0.8.0           py310hd52fb3e_4    conda-forge
 pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
 pyrsistent                0.19.3          py310h1fa729e_0    conda-forge
 pysocks                   1.7.1              pyha2e5f31_6    conda-forge
 pytest                    7.2.2              pyhd8ed1ab_0    conda-forge
 pytest-benchmark          4.0.0              pyhd8ed1ab_0    conda-forge
 pytest-cases              3.6.14             pyhd8ed1ab_0    conda-forge
 pytest-cov                4.0.0              pyhd8ed1ab_0    conda-forge
 pytest-xdist              3.2.1              pyhd8ed1ab_0    conda-forge
 python                    3.10.10         he550d4f_0_cpython    conda-forge
 python-confluent-kafka    1.9.2           py310h5764c6d_2    conda-forge
 python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
 python-fastjsonschema     2.16.3             pyhd8ed1ab_0    conda-forge
 python-jose               3.3.0              pyh6c4a22f_1    conda-forge
 python-json-logger        2.0.7              pyhd8ed1ab_0    conda-forge
 python-snappy             0.6.1           py310hcee4d7c_0    conda-forge
 python-xxhash             3.2.0           py310h1fa729e_0    conda-forge
 python_abi                3.10                    3_cp310    conda-forge
 pytorch                   1.11.0             py3.10_cpu_0    pytorch
 pytorch-mutex             1.0                         cpu    pytorch
 pytz                      2023.2             pyhd8ed1ab_0    conda-forge
 pywin32-on-windows        0.1.0              pyh1179c8e_3    conda-forge
 pyyaml                    6.0             py310h5764c6d_5    conda-forge
 pyzmq                     25.0.2          py310h059b190_0    conda-forge
 re2                       2023.02.02           hcb278e6_0    conda-forge
 readline                  8.2                  h8228510_1    conda-forge
 recommonmark              0.7.1              pyhd8ed1ab_0    conda-forge
 regex                     2023.3.23       py310h1fa729e_0    conda-forge
 requests                  2.28.2             pyhd8ed1ab_0    conda-forge
 responses                 0.21.0             pyhd8ed1ab_0    conda-forge
 rfc3339-validator         0.1.4              pyhd8ed1ab_0    conda-forge
 rfc3986-validator         0.1.1              pyh9f0ad1d_0    conda-forge
 rhash                     1.4.3                h166bdaf_0    conda-forge
 rmm                       23.06.00a       cuda11_py310_230323_gbc9155e9_0    rapidsai-nightly
 rsa                       4.9                pyhd8ed1ab_0    conda-forge
 s2n                       1.3.41               h3358134_0    conda-forge
 s3fs                      2023.3.0           pyhd8ed1ab_0    conda-forge
 s3transfer                0.6.0              pyhd8ed1ab_0    conda-forge
 sacremoses                0.0.53             pyhd8ed1ab_0    conda-forge
 sarif-om                  1.0.4              pyhd8ed1ab_0    conda-forge
 scikit-build              0.16.7             pyh56297ac_0    conda-forge
 scipy                     1.10.1          py310h8deb116_0    conda-forge
 sed                       4.8                  he412f7d_0    conda-forge
 send2trash                1.8.0              pyhd8ed1ab_0    conda-forge
 setuptools                67.6.0             pyhd8ed1ab_0    conda-forge
 six                       1.16.0             pyh6c4a22f_0    conda-forge
 snappy                    1.1.10               h9fff704_0    conda-forge
 sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
 snowballstemmer           2.2.0              pyhd8ed1ab_0    conda-forge
 sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
 soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
 spdlog                    1.11.0               h9b3ece8_1    conda-forge
 sphinx                    5.3.0              pyhd8ed1ab_0    conda-forge
 sphinx-autobuild          2021.3.14          pyhd8ed1ab_0    conda-forge
 sphinx-copybutton         0.5.1              pyhd8ed1ab_0    conda-forge
 sphinx-markdown-tables    0.0.17             pyh6c4a22f_0    conda-forge
 sphinxcontrib-applehelp   1.0.4              pyhd8ed1ab_0    conda-forge
 sphinxcontrib-devhelp     1.0.2                      py_0    conda-forge
 sphinxcontrib-htmlhelp    2.0.1              pyhd8ed1ab_0    conda-forge
 sphinxcontrib-jsmath      1.0.1                      py_0    conda-forge
 sphinxcontrib-qthelp      1.0.3                      py_0    conda-forge
 sphinxcontrib-serializinghtml 1.1.5              pyhd8ed1ab_2    conda-forge
 sphinxcontrib-websupport  1.2.4              pyhd8ed1ab_1    conda-forge
 sqlalchemy                1.4.46          py310h1fa729e_0    conda-forge
 sshpubkeys                3.3.1              pyhd8ed1ab_0    conda-forge
 stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
 streamz                   0.6.4              pyh6c4a22f_0    conda-forge
 sysroot_linux-64          2.17                h4a8ded7_13    conda-forge
 tabulate                  0.9.0              pyhd8ed1ab_1    conda-forge
 tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
 terminado                 0.17.1             pyh41d4057_0    conda-forge
 tinycss2                  1.2.1              pyhd8ed1ab_0    conda-forge
 tk                        8.6.12               h27826a3_0    conda-forge
 tokenizers                0.13.1          py310h633acb5_2    conda-forge
 toml                      0.10.2             pyhd8ed1ab_0    conda-forge
 tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
 toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
 tornado                   6.2             py310h5764c6d_1    conda-forge
 tqdm                      4.65.0             pyhd8ed1ab_1    conda-forge
 traitlets                 5.9.0              pyhd8ed1ab_0    conda-forge
 transformers              4.24.0             pyhd8ed1ab_0    conda-forge
 typing-extensions         4.5.0                hd8ed1ab_0    conda-forge
 typing_extensions         4.5.0              pyha770c72_0    conda-forge
 tzdata                    2023b                h71feb2d_0    conda-forge
 ucx                       1.14.0               h538f049_0    conda-forge
 ukkonen                   1.0.1           py310hbf28c38_3    conda-forge
 urllib3                   1.26.15            pyhd8ed1ab_0    conda-forge
 virtualenv                20.21.0            pyhd8ed1ab_0    conda-forge
 wcwidth                   0.2.6              pyhd8ed1ab_0    conda-forge
 webencodings              0.5.1                      py_1    conda-forge
 websocket-client          1.5.1              pyhd8ed1ab_0    conda-forge
 werkzeug                  2.1.2              pyhd8ed1ab_1    conda-forge
 wheel                     0.40.0             pyhd8ed1ab_0    conda-forge
 wrapt                     1.15.0          py310h1fa729e_0    conda-forge
 xmltodict                 0.13.0             pyhd8ed1ab_0    conda-forge
 xorg-libxau               1.0.9                h7f98852_0    conda-forge
 xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
 xxhash                    0.8.1                h0b41bf4_0    conda-forge
 xz                        5.2.6                h166bdaf_0    conda-forge
 yaml                      0.2.5                h7f98852_2    conda-forge
 yarl                      1.8.2           py310h5764c6d_0    conda-forge
 zeromq                    4.3.4                h9c3ff4c_1    conda-forge
 zict                      2.2.0              pyhd8ed1ab_0    conda-forge
 zipp                      3.15.0             pyhd8ed1ab_0    conda-forge
 zlib                      1.2.13               h166bdaf_4    conda-forge
 zstd                      1.5.2                h3eb15da_6    conda-forge

Additional context
Add any other context about the problem here.
@galipremsagar galipremsagar added bug Something isn't working Needs Triage Need team to review and classify cuIO cuIO issue labels Mar 29, 2023
@GregoryKimball
Copy link
Contributor

GregoryKimball commented Mar 29, 2023 •
edited
Loading

Thank you @galipremsagar for raising this issue. Using the 23.02 release image, I reproduced your issue with List types:

cudf.DataFrame({'a':[[1,2],[3],[4, None], None, [1,2]]*2000}).to_json('tmp.j', engine='cudf', orient='records', lines=True)

RuntimeError: for_each: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered

I also identified a potentially related error with a simpler list column:

cudf.DataFrame({'a':[[1]]*10000}).to_json('tmp.j', engine='cudf', orient='records', lines=True)

RuntimeError: CUDF failure at: /opt/conda/conda-bld/work/cpp/src/column/column_factories.cpp:82: Column size cannot be negative.

and also...

cudf.DataFrame({'a':[[1]]*100}).to_json('tmp.j', engine='cudf', orient='records', lines=True)

MemoryError: std::bad_alloc: CUDA error at: /opt/conda/envs/rapids/include/rmm/mr/device/cuda_memory_resource.hpp

@GregoryKimball GregoryKimball added 1 - On Deck To be worked on next libcudf Affects libcudf (C++/CUDA) code. and removed Needs Triage Need team to review and classify labels Mar 29, 2023
@ttnghia
Copy link
Contributor

ttnghia commented Mar 29, 2023

This appears to not be an OOM, as I'm able to write a similarly sized string column

I believe that strings column and lists column are processed differently. So you may still have OOM with lists while not with strings.

cudaErrorIllegalAddress may be due to either OOM (due to overflow of the size used to allocate memory) or indeed illegal access to some out-of-bound address. Of course we need more investigation.

@karthikeyann
Copy link
Contributor

Thanks @ttnghia
The issue is in list to string conversion of chunked json writer.
Minimal repro: (Run with compute-sanitizer --tool memcheck).

import cudf
gdf = cudf.DataFrame({"lists": [[1, 2, 3], [4]]*5})
gdf.to_json("cudf.json", engine="cudf", orient='records', lines=True, rows_per_chunk=8)

@karthikeyann karthikeyann mentioned this issue Apr 10, 2023
Merged
3 tasks
rapids-bot bot pushed a commit that referenced this issue Apr 11, 2023
@karthikeyann
Fixes sliced list and struct column bug in JSON chunked writer (#13108)
5638d44 
Fixes the OOM access error while using chunked JSON writer on list columns.
The issue is present in struct columns also, which is fixed in this change.
Fixes #13030

Authors:
  - Karthikeyan (https://github.com/karthikeyann)
  - GALI PREM SAGAR (https://github.com/galipremsagar)

Approvers:
  - Nghia Truong (https://github.com/ttnghia)
  - David Wendt (https://github.com/davidwendt)

URL: #13108
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@karthikeyann karthikeyann

Labels
1 - On Deck To be worked on next bug Something isn't working cuIO cuIO issue libcudf Affects libcudf (C++/CUDA) code.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fixes sliced list and struct column bug in JSON chunked writer karthikeyann/cudf
4 participants
@karthikeyann @ttnghia @galipremsagar @GregoryKimball