Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Nightly CI issue: kerneldensity cupy fusekernels fail hypothesis flaky test in CUDA 12.0.1 job with cupy compilation error. #6173

Open
dantegd opened this issue Dec 10, 2024 · 0 comments
Open

[BUG] Nightly CI issue: kerneldensity cupy fusekernels fail hypothesis flaky test in CUDA 12.0.1 job with cupy compilation error. #6173

dantegd opened this issue Dec 10, 2024 · 0 comments
Labels
? - Needs Triage Need team to review and classify bug Something isn't working

Comments

@dantegd
Copy link
Member

dantegd commented Dec 10, 2024

Nightly jobs show the following error:

E               hypothesis.errors.FlakyReplay: Inconsistent results from replaying a test case!
E                 last: INTERESTING from CompileException at /opt/conda/envs/test/lib/python3.11/site-packages/cupy/cuda/compiler.py:711
E                   context: NVRTCError at cupy_backends/cuda/libs/nvrtc.pyx:53
E                       context: TypeError at cupy/_core/fusion.pyx:554
E                 this: INTERESTING from CompileException at /opt/conda/envs/test/lib/python3.11/site-packages/cupy/cuda/compiler.py:711
E                   context: NVRTCError at cupy_backends/cuda/libs/nvrtc.pyx:53

Digging deeper this is coming from CuPy:

E           cupy.cuda.compiler.CompileException: /tmp/tmpueuqs7aw/67091474e9b3014299476429db97e8d39a66649f.cubin.cu(5): catastrophic error: cannot open source file "cooperative_groups.h"

Using the same containers outside of CI this does not reproduce currently.
@dantegd dantegd added ? - Needs Triage Need team to review and classify bug Something isn't working labels Dec 10, 2024
@dantegd dantegd mentioned this issue Dec 16, 2024
Merged
rapids-bot bot pushed a commit that referenced this issue Dec 17, 2024
@dantegd
Skip flaky test of kernel_density in nightly job of CUDA 12.0.1 (#6184)
e33f467 
PR skips this test in the relevant nightly job affected by issue #6173 while we keep looking further into it.

Authors:
  - Dante Gama Dessavre (https://github.com/dantegd)

Approvers:
  - William Hicks (https://github.com/wphicks)

URL: #6184
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
? - Needs Triage Need team to review and classify bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@dantegd