Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] CUDA_STREAM_TEST segfaults in release build #932

Closed
rongou opened this issue Dec 8, 2021 · 5 comments · Fixed by #934
Closed

[BUG] CUDA_STREAM_TEST segfaults in release build #932

rongou opened this issue Dec 8, 2021 · 5 comments · Fixed by #934
Assignees
@harrism
Labels
? - Needs Triage Need team to review and classify bug Something isn't working

Comments

@rongou
Copy link
Contributor

rongou commented Dec 8, 2021

Describe the bug
In a release build the tests CUDA_STREAM_TEST and CUDA_STREAM_PTDS_TEST segfault.

Steps/Code to reproduce bug

cmake .. -DCMAKE_BUILD_TYPE=Release
make
./gtests/CUDA_STREAM_TEST
Running main() from gmock_main.cc
[==========] Running 12 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from CudaStreamTest
[ RUN      ] CudaStreamTest.Equality
[       OK ] CudaStreamTest.Equality (88 ms)
[ RUN      ] CudaStreamTest.MoveConstructor
[       OK ] CudaStreamTest.MoveConstructor (0 ms)
[ RUN      ] CudaStreamTest.TestSyncNoThrow
Segmentation fault (core dumped)

I think the issue is TestSyncNoThrow calls cudaStreamDestroy, while the rmm::cuda_stream destructor also calls cudaStreamDestroy.

Expected behavior
Tests should not segfault in release build.

Environment details (please complete the following information):

  • Environment location: Bare-metal
  • Method of RMM install: from source
@rongou rongou added bug Something isn't working ? - Needs Triage Need team to review and classify labels Dec 8, 2021
@harrism
Copy link
Member

harrism commented Dec 9, 2021 •
edited
Loading

@rongou I'm unable to repro with CUDA 11.5. What are your system details?

@harrism
Copy link
Member

harrism commented Dec 9, 2021

Calling cudaStreamDestroy on an already destroyed stream should cause a CUDA error, not a segfault.

@rongou
Copy link
Contributor Author

rongou commented Dec 9, 2021

I'm on CUDA 11.5.1. Here is the output from ./print_env.sh:

**git***
commit bd5aa5d9046eeb492ab412307c00644f622e3de6 (HEAD -> branch-22.02, upstream/branch-22.02, origin/branch-22.02)
Author: Mark Harris <[email protected]>
Date:   Tue Dec 7 09:09:11 2021 +1100

    Fix failing tracking test (#929)
    
    Fixes #928.
    
    #920 introduced a new test for the `tracking_resource_adaptor::log_outstanding_allocations`, but that function only logs when `SPDLOG_ACTIVE_LEVEL <= SPDLOG_LEVEL_DEBUG`, so the test needs to be gated on that condition as well.
    
    Authors:
      - Mark Harris (https://github.com/harrism)
    
    Approvers:
      - Rong Ou (https://github.com/rongou)
      - Vyas Ramasubramani (https://github.com/vyasr)
      - Bradley Dice (https://github.com/bdice)
    
    URL: https://github.com/rapidsai/rmm/pull/929

***OS Information***
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.3 LTS"
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
Linux rou 5.4.0-80-generic #90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

***GPU Information***
Thu Dec  9 09:09:25 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 495.44       Driver Version: 495.44       CUDA Version: 11.5     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA TITAN V      Off  | 00000000:01:00.0  On |                  N/A |
| 29%   42C    P8    27W / 250W |    867MiB / 12057MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Quadro RTX 6000     Off  | 00000000:21:00.0 Off |                  Off |
| 33%   32C    P8    29W / 260W |      1MiB / 24220MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      3557      G   /usr/lib/xorg/Xorg                 96MiB |
|    0   N/A  N/A      4838      G   /usr/lib/xorg/Xorg                506MiB |
|    0   N/A  N/A      4983      G   /usr/bin/gnome-shell               82MiB |
|    0   N/A  N/A    827176      G   ...AAAAAAAAA= --shared-files       27MiB |
|    0   N/A  N/A    926510      G   ...AAAAAAAAA= --shared-files       32MiB |
|    0   N/A  N/A   2723893      G   .../debug.log --shared-files       29MiB |
|    0   N/A  N/A   3716350      G   ...AAAAAAAAA= --shared-files       67MiB |
+-----------------------------------------------------------------------------+

***CPU***
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   43 bits physical, 48 bits virtual
CPU(s):                          48
On-line CPU(s) list:             0-47
Thread(s) per core:              2
Core(s) per socket:              24
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       AuthenticAMD
CPU family:                      23
Model:                           49
Model name:                      AMD Ryzen Threadripper 3960X 24-Core Processor
Stepping:                        0
Frequency boost:                 enabled
CPU MHz:                         2195.203
CPU max MHz:                     3800.0000
CPU min MHz:                     2200.0000
BogoMIPS:                        7586.08
Virtualization:                  AMD-V
L1d cache:                       768 KiB
L1i cache:                       768 KiB
L2 cache:                        12 MiB
L3 cache:                        128 MiB
NUMA node0 CPU(s):               0-47
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full AMD retpoline, IBPB conditional, STIBP conditional, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonst
                                 op_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a 
                                 misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate sme ssbd mba sev ibpb stibp vmmcall fsgsbas
                                 e bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr w
                                 bnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca

***CMake***
/home/rou/.miniconda3/envs/cudf_dev/bin/cmake
cmake version 3.21.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).

***g++***
/usr/local/bin/g++
g++ (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


***nvcc***
/usr/local/bin/nvcc
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Thu_Nov_18_09:45:30_PST_2021
Cuda compilation tools, release 11.5, V11.5.119
Build cuda_11.5.r11.5/compiler.30672275_0

***Python***
/home/rou/.miniconda3/envs/cudf_dev/bin/python
Python 3.8.12

***Environment Variables***
PATH                            : /home/rou/.miniconda3/envs/cudf_dev/bin:/home/rou/.miniconda3/condabin:/home/rou/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/cuda/bin
LD_LIBRARY_PATH                 : 
NUMBAPRO_NVVM                   : 
NUMBAPRO_LIBDEVICE              : 
CONDA_PREFIX                    : /home/rou/.miniconda3/envs/cudf_dev
PYTHON_PATH                     : 

***conda packages***
/home/rou/.miniconda3/condabin/conda
# packages in environment at /home/rou/.miniconda3/envs/cudf_dev:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                      1_llvm    conda-forge
abseil-cpp                20210324.2           h9c3ff4c_0    conda-forge
alabaster                 0.7.12                     py_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
argon2-cffi               21.1.0           py38h497a2fe_2    conda-forge
arrow-cpp                 5.0.0           py38hcdf9ef9_19_cuda    conda-forge
arrow-cpp-proc            3.0.0                      cuda    conda-forge
async_generator           1.10                       py_0    conda-forge
attrs                     21.2.0             pyhd8ed1ab_0    conda-forge
aws-c-auth                0.6.8                hadad3cd_1    conda-forge
aws-c-cal                 0.5.12               h70efedd_7    conda-forge
aws-c-common              0.6.17               h7f98852_0    conda-forge
aws-c-compression         0.2.14               h7c7754b_7    conda-forge
aws-c-event-stream        0.2.7               hd2be095_32    conda-forge
aws-c-http                0.6.10               h416565a_3    conda-forge
aws-c-io                  0.10.14              he836878_0    conda-forge
aws-c-mqtt                0.7.10               h885097b_0    conda-forge
aws-c-s3                  0.1.29               h8d70ed6_0    conda-forge
aws-c-sdkutils            0.1.1                h7c7754b_4    conda-forge
aws-checksums             0.1.12               h7c7754b_6    conda-forge
aws-crt-cpp               0.17.10              h6ab17b9_5    conda-forge
aws-sdk-cpp               1.9.160              h36ff4c5_0    conda-forge
babel                     2.9.1              pyh44b312d_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports-zoneinfo        0.2.1                    pypi_0    pypi
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.10.0             pyha770c72_0    conda-forge
binutils_impl_linux-64    2.36.1               h193b22a_2    conda-forge
black                     19.10b0                    py_4    conda-forge
bleach                    4.1.0              pyhd8ed1ab_0    conda-forge
bokeh                     2.4.2            py38h578d9bd_0    conda-forge
brotlipy                  0.7.0           py38h497a2fe_1003    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2021.10.8            ha878542_0    conda-forge
cachetools                4.2.4              pyhd8ed1ab_0    conda-forge
certifi                   2021.10.8        py38h578d9bd_1    conda-forge
cffi                      1.15.0           py38h3931269_0    conda-forge
cfgv                      3.3.1              pyhd8ed1ab_0    conda-forge
charset-normalizer        2.0.9              pyhd8ed1ab_0    conda-forge
clang                     11.1.0               ha770c72_1    conda-forge
clang-11                  11.1.0          default_ha53f305_1    conda-forge
clang-tools               11.1.0          default_ha53f305_1    conda-forge
clangxx                   11.1.0          default_ha53f305_1    conda-forge
click                     8.0.3            py38h578d9bd_1    conda-forge
cloudpickle               2.0.0              pyhd8ed1ab_0    conda-forge
cmake                     3.21.3               h8897547_0    conda-forge
cmake_setuptools          0.1.3                      py_0    rapidsai
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
commonmark                0.9.1                      py_0    conda-forge
cryptography              36.0.0           py38h3e25421_0    conda-forge
cudatoolkit               11.5.0               h36ae40a_9    nvidia
cupy                      9.6.0            py38h177b0fd_0    conda-forge
cyrus-sasl                2.1.27               h230043b_5    conda-forge
cython                    0.29.25          py38h709712a_0    conda-forge
cytoolz                   0.11.2           py38h497a2fe_1    conda-forge
dask                      2021.11.2+24.gecddc8df          pypi_0    pypi
dataclasses               0.8                pyhc8e2a94_3    conda-forge
debugpy                   1.5.1            py38h709712a_0    conda-forge
decorator                 5.1.0              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
distlib                   0.3.4              pyhd8ed1ab_0    conda-forge
distributed               2021.11.2+22.g491e28a2          pypi_0    pypi
dlpack                    0.5                  h9c3ff4c_0    conda-forge
docutils                  0.17.1           py38h578d9bd_1    conda-forge
double-conversion         3.1.6                h9c3ff4c_0    conda-forge
editdistance-s            1.0.0            py38h1fd1430_2    conda-forge
entrypoints               0.3             pyhd8ed1ab_1003    conda-forge
execnet                   1.9.0              pyhd8ed1ab_0    conda-forge
expat                     2.4.1                h9c3ff4c_0    conda-forge
fastavro                  1.4.7            py38h497a2fe_1    conda-forge
fastrlock                 0.8              py38h709712a_1    conda-forge
filelock                  3.4.0              pyhd8ed1ab_0    conda-forge
flake8                    3.8.3                      py_1    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
fsspec                    2021.11.1          pyhd8ed1ab_0    conda-forge
future                    0.18.2           py38h578d9bd_4    conda-forge
gcc_impl_linux-64         11.2.0              h82a94d6_11    conda-forge
gettext                   0.19.8.1          h73d1719_1008    conda-forge
gflags                    2.2.2             he1b5a44_1004    conda-forge
glog                      0.5.0                h48cff8f_0    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
grpc-cpp                  1.42.0               h7e358d9_0    conda-forge
heapdict                  1.0.1                      py_0    conda-forge
huggingface_hub           0.2.1              pyhd8ed1ab_0    conda-forge
hypothesis                6.31.0             pyhd8ed1ab_0    conda-forge
identify                  2.3.7              pyhd8ed1ab_0    conda-forge
idna                      3.1                pyhd3deb0d_0    conda-forge
imagesize                 1.3.0              pyhd8ed1ab_0    conda-forge
importlib-metadata        4.8.2            py38h578d9bd_0    conda-forge
importlib_metadata        4.8.2                hd8ed1ab_0    conda-forge
importlib_resources       5.4.0              pyhd8ed1ab_0    conda-forge
iniconfig                 1.1.1              pyh9f0ad1d_0    conda-forge
ipykernel                 6.6.0            py38he5a9106_0    conda-forge
ipython                   7.30.1           py38h578d9bd_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
isort                     5.6.4                      py_0    conda-forge
jbig                      2.1               h7f98852_2003    conda-forge
jedi                      0.18.1           py38h578d9bd_0    conda-forge
jinja2                    3.0.3              pyhd8ed1ab_0    conda-forge
joblib                    1.1.0              pyhd8ed1ab_0    conda-forge
jpeg                      9d                   h36c2ea0_0    conda-forge
jsonschema                4.2.1              pyhd8ed1ab_1    conda-forge
jupyter_client            7.1.0              pyhd8ed1ab_0    conda-forge
jupyter_core              4.9.1            py38h578d9bd_1    conda-forge
jupyterlab_pygments       0.1.2              pyh9f0ad1d_0    conda-forge
kernel-headers_linux-64   2.6.32              he073ed8_15    conda-forge
krb5                      1.19.2               hcc1bbae_3    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
lerc                      3.0                  h9c3ff4c_0    conda-forge
libblas                   3.9.0            12_linux64_mkl    conda-forge
libbrotlicommon           1.0.9                h7f98852_6    conda-forge
libbrotlidec              1.0.9                h7f98852_6    conda-forge
libbrotlienc              1.0.9                h7f98852_6    conda-forge
libcblas                  3.9.0            12_linux64_mkl    conda-forge
libclang-cpp11.1          11.1.0          default_ha53f305_1    conda-forge
libcurl                   7.80.0               h2574ce0_0    conda-forge
libdeflate                1.8                  h7f98852_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               h9b69904_4    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-devel_linux-64     11.2.0              h0952999_11    conda-forge
libgcc-ng                 11.2.0              h1d223b6_11    conda-forge
libgcrypt                 1.9.4                h7f98852_0    conda-forge
libgomp                   11.2.0              h1d223b6_11    conda-forge
libgpg-error              1.42                 h9c3ff4c_0    conda-forge
libgsasl                  1.10.0               h5b4c23d_0    conda-forge
liblapack                 3.9.0            12_linux64_mkl    conda-forge
libllvm11                 11.1.0               hf817b99_2    conda-forge
libnghttp2                1.43.0               h812cca2_1    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libntlm                   1.4               h7f98852_1002    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libprotobuf               3.18.1               h780b84a_0    conda-forge
librdkafka                1.7.0                hc49e61c_0    conda-forge
librmm                    22.02.00a211209 cuda11_gbd5aa5d_20    rapidsai-nightly
libsanitizer              11.2.0              he4da1e4_11    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libssh2                   1.10.0               ha56f1ee_2    conda-forge
libstdcxx-ng              11.2.0              he4da1e4_11    conda-forge
libthrift                 0.15.0               he6d91bd_1    conda-forge
libtiff                   4.3.0                h6f004c6_2    conda-forge
libutf8proc               2.6.1                h7f98852_0    conda-forge
libuv                     1.42.0               h7f98852_0    conda-forge
libwebp-base              1.2.1                h7f98852_0    conda-forge
libzlib                   1.2.11            h36c2ea0_1013    conda-forge
llvm-openmp               12.0.1               h4bd325d_1    conda-forge
llvmlite                  0.37.0           py38h4630a5e_1    conda-forge
locket                    0.2.0                      py_2    conda-forge
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
markdown                  3.3.6              pyhd8ed1ab_0    conda-forge
markupsafe                2.0.1            py38h497a2fe_1    conda-forge
matplotlib-inline         0.1.3              pyhd8ed1ab_0    conda-forge
mccabe                    0.6.1                      py_1    conda-forge
mimesis                   4.0.0              pyh9f0ad1d_0    conda-forge
mistune                   0.8.4           py38h497a2fe_1005    conda-forge
mkl                       2021.4.0           h8d4b97c_729    conda-forge
more-itertools            8.12.0             pyhd8ed1ab_0    conda-forge
msgpack-python            1.0.3            py38h1fd1430_0    conda-forge
mypy                      0.782                      py_0    conda-forge
mypy_extensions           0.4.3            py38h578d9bd_4    conda-forge
nbclient                  0.5.9              pyhd8ed1ab_0    conda-forge
nbconvert                 6.3.0            py38h578d9bd_1    conda-forge
nbformat                  5.1.3              pyhd8ed1ab_0    conda-forge
nbsphinx                  0.8.7              pyhd8ed1ab_0    conda-forge
ncurses                   6.2                  h58526e2_4    conda-forge
nest-asyncio              1.5.4              pyhd8ed1ab_0    conda-forge
ninja                     1.10.2               h4bd325d_1    conda-forge
nodeenv                   1.6.0              pyhd8ed1ab_0    conda-forge
notebook                  6.4.6              pyha770c72_0    conda-forge
numba                     0.54.1           py38h4bf6c61_0    conda-forge
numpy                     1.20.3           py38h9894fe3_1    conda-forge
numpydoc                  1.1.0                      py_1    conda-forge
nvtx                      0.2.3            py38h497a2fe_1    conda-forge
olefile                   0.46               pyh9f0ad1d_1    conda-forge
openjpeg                  2.4.0                hb52868f_1    conda-forge
openssl                   1.1.1l               h7f98852_0    conda-forge
orc                       1.7.1                h68e2c4e_0    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandas                    1.3.4            py38h43a58ef_1    conda-forge
pandoc                    1.19.2                        0    conda-forge
pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
parquet-cpp               1.5.1                         2    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
partd                     1.2.0              pyhd8ed1ab_0    conda-forge
pathspec                  0.9.0              pyhd8ed1ab_0    conda-forge
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    8.4.0            py38h8e6f84c_0    conda-forge
pip                       21.3.1             pyhd8ed1ab_0    conda-forge
pluggy                    1.0.0            py38h578d9bd_2    conda-forge
pre-commit                2.16.0           py38h578d9bd_0    conda-forge
prometheus_client         0.12.0             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.23             pyha770c72_0    conda-forge
protobuf                  3.18.1           py38h709712a_0    conda-forge
psutil                    5.8.0            py38h497a2fe_2    conda-forge
ptxcompiler               0.1.0            py38hb739d79_0    rapidsai
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
py                        1.11.0             pyh6c4a22f_0    conda-forge
py-cpuinfo                8.0.0              pyhd8ed1ab_0    conda-forge
pyarrow                   5.0.0           py38ha746e9d_19_cuda    conda-forge
pycodestyle               2.6.0              pyh9f0ad1d_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pydata-sphinx-theme       0.7.2              pyhd8ed1ab_0    conda-forge
pydocstyle                6.1.1              pyhd8ed1ab_0    conda-forge
pyflakes                  2.2.0              pyh9f0ad1d_0    conda-forge
pygments                  2.10.0             pyhd8ed1ab_0    conda-forge
pyopenssl                 21.0.0             pyhd8ed1ab_0    conda-forge
pyorc                     0.5.0                    pypi_0    pypi
pyparsing                 3.0.6              pyhd8ed1ab_0    conda-forge
pyrsistent                0.18.0           py38h497a2fe_0    conda-forge
pysocks                   1.7.1            py38h578d9bd_4    conda-forge
pytest                    6.2.5            py38h578d9bd_1    conda-forge
pytest-benchmark          3.4.1              pyhd8ed1ab_0    conda-forge
pytest-forked             1.3.0              pyhd3deb0d_0    conda-forge
pytest-xdist              2.4.0              pyhd8ed1ab_0    conda-forge
python                    3.8.12          hb7a2778_2_cpython    conda-forge
python-confluent-kafka    1.7.0            py38h497a2fe_2    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-snappy             0.6.0            py38h49bdff1_1    conda-forge
python_abi                3.8                      2_cp38    conda-forge
pytorch                   1.10.0          cpu_py38h1ee18c8_0    conda-forge
pytz                      2021.3             pyhd8ed1ab_0    conda-forge
pyyaml                    6.0              py38h497a2fe_3    conda-forge
pyzmq                     22.3.0           py38h2035c66_1    conda-forge
rapidjson                 1.1.0             he1b5a44_1002    conda-forge
re2                       2021.11.01           h9c3ff4c_0    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
recommonmark              0.7.1              pyhd8ed1ab_0    conda-forge
regex                     2021.11.10       py38h497a2fe_0    conda-forge
requests                  2.26.0             pyhd8ed1ab_1    conda-forge
rhash                     1.4.1                h7f98852_0    conda-forge
rmm                       22.02.00a211209 cuda11_py38_gbd5aa5d_20_has_cma    rapidsai-nightly
s2n                       1.3.0                h9b69904_0    conda-forge
sacremoses                0.0.46             pyhd8ed1ab_0    conda-forge
send2trash                1.8.0              pyhd8ed1ab_0    conda-forge
setuptools                59.4.0           py38h578d9bd_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sleef                     3.5.1                h9b69904_2    conda-forge
snappy                    1.1.8                he1b5a44_3    conda-forge
snowballstemmer           2.2.0              pyhd8ed1ab_0    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.3.1              pyhd8ed1ab_0    conda-forge
spdlog                    1.8.5                h4bd325d_0    conda-forge
sphinx                    4.3.1              pyh6c4a22f_0    conda-forge
sphinx-copybutton         0.4.0              pyhd8ed1ab_0    conda-forge
sphinx-markdown-tables    0.0.15             pyhd3deb0d_0    conda-forge
sphinxcontrib-applehelp   1.0.2                      py_0    conda-forge
sphinxcontrib-devhelp     1.0.2                      py_0    conda-forge
sphinxcontrib-htmlhelp    2.0.0              pyhd8ed1ab_0    conda-forge
sphinxcontrib-jsmath      1.0.1                      py_0    conda-forge
sphinxcontrib-qthelp      1.0.3                      py_0    conda-forge
sphinxcontrib-serializinghtml 1.1.5              pyhd8ed1ab_1    conda-forge
sphinxcontrib-websupport  1.2.4              pyhd8ed1ab_1    conda-forge
sqlite                    3.37.0               h9cd32fc_0    conda-forge
streamz                   0.6.3              pyh6c4a22f_0    conda-forge
sysroot_linux-64          2.12                he073ed8_15    conda-forge
tbb                       2021.4.0             h4bd325d_1    conda-forge
tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
terminado                 0.12.1           py38h578d9bd_1    conda-forge
testpath                  0.5.0              pyhd8ed1ab_0    conda-forge
tk                        8.6.11               h27826a3_1    conda-forge
tokenizers                0.10.3           py38hb63a372_1    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
toolz                     0.11.2             pyhd8ed1ab_0    conda-forge
tornado                   6.1              py38h497a2fe_2    conda-forge
tqdm                      4.62.3             pyhd8ed1ab_0    conda-forge
traitlets                 5.1.1              pyhd8ed1ab_0    conda-forge
transformers              4.10.3             pyhd8ed1ab_0    conda-forge
typed-ast                 1.4.3            py38h497a2fe_1    conda-forge
typing-extensions         4.0.1                hd8ed1ab_0    conda-forge
typing_extensions         4.0.1              pyha770c72_0    conda-forge
urllib3                   1.26.7             pyhd8ed1ab_0    conda-forge
virtualenv                20.4.7           py38h578d9bd_1    conda-forge
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
webencodings              0.5.1                      py_1    conda-forge
wheel                     0.37.0             pyhd8ed1ab_1    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
yaml                      0.2.5                h516909a_0    conda-forge
zeromq                    4.3.4                h9c3ff4c_1    conda-forge
zict                      2.0.0                      py_0    conda-forge
zipp                      3.6.0              pyhd8ed1ab_0    conda-forge
zlib                      1.2.11            h36c2ea0_1013    conda-forge
zstd                      1.5.0                ha95c52a_0    conda-forge

@rongou
Copy link
Contributor Author

rongou commented Dec 9, 2021

At least on my machine, this seems reproducible:

$ cat foo.cu
int main()
{
  auto* stream = new cudaStream_t;
  cudaStreamCreate(stream);
  cudaStreamDestroy(*stream);
  cudaStreamDestroy(*stream);
}
$ nvcc foo.cu
$ ./a.out
Segmentation fault (core dumped)

@harrism harrism mentioned this issue Dec 13, 2021
Merged
@harrism
Copy link
Member

harrism commented Dec 13, 2021

Wasn't able to repro but I came up with a better way to test in #934. Please review @rongou. Thanks!

@rapids-bot rapids-bot bot closed this as completed in #934 Dec 14, 2021
rapids-bot bot pushed a commit that referenced this issue Dec 14, 2021
@harrism
Fix failing stream test with a debug-only death test (#934)
cd5f806 
Fixes #932.  Adds a debug-build-only death test for `cuda_stream::synchronize_no_throw` and adds a likely never to fail trivial test of the function in all builds.   This way we get coverage of the function in all builds, and can safely test its assertion in debug builds.

Authors:
  - Mark Harris (https://github.com/harrism)

Approvers:
  - Rong Ou (https://github.com/rongou)
  - Conor Hoekstra (https://github.com/codereport)

URL: #934
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@harrism harrism

Labels
? - Needs Triage Need team to review and classify bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix failing stream test with a debug-only death test harrism/rmm
2 participants
@rongou @harrism