Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add environment variables for I/O thread pool and slice sizes #10218

Merged
merged 3 commits into from
Feb 4, 2022

Conversation

vuule
Copy link
Contributor

@vuule vuule commented Feb 4, 2022

Modified getenv_or to allow the default value of any type.
Adds environment variables:

  • LIBCUDF_CUFILE_THREAD_COUNT: integer, number of threads in the pool used for I/O (default 16);
  • LIBCUDF_CUFILE_SLICE_SIZE: integer, maximum size of cuFile reads/writes in bytes (default 4MB).

@vuule vuule added cuIO cuIO issue Performance Performance related issue improvement Improvement / enhancement to an existing function labels Feb 4, 2022
@vuule vuule self-assigned this Feb 4, 2022
@github-actions github-actions bot added the libcudf Affects libcudf (C++/CUDA) code. label Feb 4, 2022
@codecov
Copy link

codecov bot commented Feb 4, 2022

Codecov Report

Merging #10218 (15f1daf) into branch-22.04 (a7d88cd) will increase coverage by 0.05%.
The diff coverage is 0.00%.

❗ Current head 15f1daf differs from pull request most recent head 71566b7. Consider uploading reports for the commit 71566b7 to get more accurate results

Impacted file tree graph

@@               Coverage Diff                @@
##           branch-22.04   #10218      +/-   ##
================================================
+ Coverage         10.42%   10.47%   +0.05%     
================================================
  Files               119      122       +3     
  Lines             20603    20487     -116     
================================================
- Hits               2148     2147       -1     
+ Misses            18455    18340     -115     
Impacted Files Coverage Δ
python/cudf/cudf/_fuzz_testing/fuzzer.py 0.00% <0.00%> (ø)
python/cudf/cudf/_fuzz_testing/io.py 0.00% <0.00%> (ø)
python/cudf/cudf/_fuzz_testing/main.py 0.00% <0.00%> (ø)
python/cudf/cudf/_version.py 0.00% <0.00%> (ø)
python/cudf/cudf/comm/gpuarrow.py 0.00% <0.00%> (ø)
python/cudf/cudf/core/_base_index.py 0.00% <0.00%> (ø)
python/cudf/cudf/core/column/categorical.py 0.00% <0.00%> (ø)
python/cudf/cudf/core/column/column.py 0.00% <0.00%> (ø)
python/cudf/cudf/core/column/datetime.py 0.00% <ø> (ø)
python/cudf/cudf/core/column/numerical.py 0.00% <0.00%> (ø)
... and 42 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0581975...71566b7. Read the comment docs.

@vuule vuule added the non-breaking Non-breaking change label Feb 4, 2022
@vuule vuule requested a review from devavret February 4, 2022 19:17
@vuule vuule marked this pull request as ready for review February 4, 2022 19:17
@vuule vuule requested a review from a team as a code owner February 4, 2022 19:17
@vuule vuule requested a review from robertmaynard February 4, 2022 19:17
@vuule
Copy link
Contributor Author

vuule commented Feb 4, 2022

@gpucibot merge

@rapids-bot rapids-bot bot merged commit 84ae8ab into rapidsai:branch-22.04 Feb 4, 2022
@vuule vuule deleted the fea-cufile-env-vars branch March 13, 2023 20:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cuIO cuIO issue improvement Improvement / enhancement to an existing function libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change Performance Performance related issue
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants