Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] text benchmark fails with thrust::system::system_error #10043

Closed
randerzander opened this issue Jan 13, 2022 · 0 comments · Fixed by #10090
Closed

[BUG] text benchmark fails with thrust::system::system_error #10043

randerzander opened this issue Jan 13, 2022 · 0 comments · Fixed by #10090
Assignees
@davidwendt
Labels
bug Something isn't working

Comments

@randerzander
Copy link
Contributor

Describe the bug
Running the libcudf TEXT_BENCH benchmark results in a thrust system error:

terminate called after throwing an instance of 'thrust::system::system_error'
  what():  copy_if failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered

Steps/Code to reproduce bug
Run the libcudf text benchmark

Expected behavior
The benchmark completes without error condition

Environment overview (please complete the following information)

  • Environment location: [Bare-metal, Docker, Cloud(specify cloud provider)]
    Docker FROM 22.02-cuda11.5-devel-ubuntu20.04-py3.7
  • Method of cuDF install: Docker
  • If method of install is [Docker], provide docker pull & docker run commands used
    After pulling the above nightly Docker image, I build & run the benchmark

Environment details
Please run and paste the output of the cudf/print_env.sh script here, to gather any other relevant environment details
https://gist.github.com/82542cb3e6c04b7a6b42d482f874400b

Additional context
Add any other context about the problem here.
@randerzander randerzander added bug Something isn't working Needs Triage Need team to review and classify labels Jan 13, 2022
@davidwendt davidwendt mentioned this issue Jan 20, 2022
Merged
rapids-bot bot pushed a commit that referenced this issue Jan 26, 2022
@davidwendt
Remove metadata singleton from nvtext normalizer (#10090)
85109e6 
Closes #10043 

A metadata singleton was allocated from the current rmm memory-resource inside the nvtext normalizer functions. If the memory resource is later changed, the metadata pointer may become invalidated. This PR removes the singleton pattern. 

The normalizer is used by the subword-tokenizer which is passed a vocabulary structure that is built only once and is maintained by the caller. The metadata has been added to this structure so it's lifetime can share the same scope.

The normalizer can also be called directly through the `nvtext::normalize_characters` API. Here the metadata table (size about 1MB) is now created on each call. This showed only significant performance impact on benchmarks testing a small number (<50K) of shorter strings.

Authors:
  - David Wendt (https://github.com/davidwendt)

Approvers:
  - Nghia Truong (https://github.com/ttnghia)
  - Robert Maynard (https://github.com/robertmaynard)

URL: #10090
@bdice bdice removed the Needs Triage Need team to review and classify label Mar 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@davidwendt davidwendt

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove metadata singleton from nvtext normalizer davidwendt/cudf
3 participants
@randerzander @bdice @davidwendt