Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Pairwise distances illegal memory access #4552

Closed
RAMitchell opened this issue Feb 4, 2022 · 1 comment
Closed

[BUG] Pairwise distances illegal memory access #4552

RAMitchell opened this issue Feb 4, 2022 · 1 comment
Labels
? - Needs Triage Need team to review and classify bug Something isn't working

Comments

@RAMitchell
Copy link
Contributor

Describe the bug
Illegal memory access in pairwise_distances. Looks like an int being used as an index.

Steps/Code to reproduce bug

import cupy as cp
from cuml.metrics import pairwise_kernels

col = 1
# row = 46000 # Pass
row = 47000  # Illegal memory access
rs = cp.random.RandomState(259)
X = rs.normal(size=(row, col), dtype=cp.float32)
print(X.shape[0]**2 < 2**31)

K = pairwise_kernels(X, metric='rbf')
False
CUDA call='cudaEventDestroy(event_)' at file=/home/nfs/rorym/cuml/cpp/build/_deps/raft-src/cpp/include/raft/handle.hpp line=304 failed with an illegal memory access was encountered
Traceback (most recent call last):
  File "illegal.py", line 11, in <module>
    K = pairwise_kernels(X, metric='cosine')
  File "/home/nfs/rorym/anaconda3/envs/cuml_dev/lib/python3.7/site-packages/cuml/internals/api_decorators.py", line 465, in inner_with_getters
    ret_val = func(*args, **kwargs)
  File "/home/nfs/rorym/anaconda3/envs/cuml_dev/lib/python3.7/site-packages/cuml/metrics/pairwise_kernels.py", line 303, in pairwise_kernels
    return PAIRWISE_KERNEL_FUNCTIONS[metric](X, Y, **kwds)
  File "/home/nfs/rorym/anaconda3/envs/cuml_dev/lib/python3.7/site-packages/cuml/metrics/pairwise_kernels.py", line 82, in cosine_similarity
    K= 1.0 - cp.asarray(pairwise_distances(X, Y, metric='cosine'))
  File "/home/nfs/rorym/anaconda3/envs/cuml_dev/lib/python3.7/site-packages/cuml/internals/api_decorators.py", line 465, in inner_with_getters
    ret_val = func(*args, **kwargs)
  File "cuml/metrics/pairwise_distances.pyx", line 315, in cuml.metrics.pairwise_distances.pairwise_distances
  File "cuml/raft/common/handle.pyx", line 76, in cuml.raft.common.handle.Handle.sync
RuntimeError: CUDA error at: /home/nfs/rorym/anaconda3/envs/cuml_dev/include/rmm/cuda_stream_view.hpp:81: cudaErrorIllegalAddress an illegal memory access was encountered
@RAMitchell RAMitchell added ? - Needs Triage Need team to review and classify bug Something isn't working labels Feb 4, 2022
@RAMitchell
Copy link
Contributor Author

Related:
#2459
#4105

rapids-bot bot pushed a commit to rapidsai/raft that referenced this issue Feb 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
? - Needs Triage Need team to review and classify bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant