Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] cudaMalloc and cudaFree are being called during aggregations #10080

Closed
jlowe opened this issue Jan 19, 2022 · 3 comments · Fixed by #10342
Closed

[BUG] cudaMalloc and cudaFree are being called during aggregations #10080

jlowe opened this issue Jan 19, 2022 · 3 comments · Fixed by #10342
Assignees
Labels
bug Something isn't working libcudf Affects libcudf (C++/CUDA) code. Performance Performance related issue Spark Functionality that helps Spark RAPIDS

Comments

@jlowe
Copy link
Member

jlowe commented Jan 19, 2022

Describe the bug
While examining a recent trace I noticed that within the libcudf aggregate range there are calls to cudaMalloc and cudaFree, the latter which causes a synchronization on the default stream. I attached gdb and put a breakpoint on cudaMalloc and found it was being triggered by cudf::detail::is_relationally_comparable<cudf::table_device_view> because it calls thrust::all_of without passing an execution policy. Without using the RMM policy, it will use the default CUDA allocator. Ideally it should be using rmm::exec_policy(stream) but the stream is not available to this method and would need to be passed.

Steps/Code to reproduce bug
Attach a debugger to a query using the RMM arena allocator and executes an aggregation. Place a breakpoint on cudaMalloc and execute the query and observe the breakpoint is hit in a callstack that derives from cudf::detail::is_relationally_comparable.

Expected behavior
libcudf should not trigger calls to cudaMalloc or cudaFree.

@jlowe jlowe added bug Something isn't working Needs Triage Need team to review and classify libcudf Affects libcudf (C++/CUDA) code. Performance Performance related issue Spark Functionality that helps Spark RAPIDS labels Jan 19, 2022
@harrism
Copy link
Member

harrism commented Jan 20, 2022

Yowza. That function should take an explicit stream.

@github-actions
Copy link

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

@jlowe
Copy link
Member Author

jlowe commented Feb 22, 2022

Still relevant and would like to see this fixed.

@davidwendt davidwendt self-assigned this Feb 22, 2022
rapids-bot bot pushed a commit that referenced this issue Mar 11, 2022
Closes #10080

Removes `is_relationally_comparable` function for `table_device_view` and `mutable_table_device_view`. The existing `is_relationally_comparable` can still be used with non-device `table_view` and `mutable_table_view`. 

Callers are updated as appropriate. The incorrect kernel call has been removed.

Authors:
  - David Wendt (https://github.com/davidwendt)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)
  - Jake Hemstad (https://github.com/jrhemstad)
  - Nghia Truong (https://github.com/ttnghia)

URL: #10342
@bdice bdice removed the Needs Triage Need team to review and classify label Mar 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working libcudf Affects libcudf (C++/CUDA) code. Performance Performance related issue Spark Functionality that helps Spark RAPIDS
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants