Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use kvikIO as the default IO backend #12574

Merged
merged 8 commits into from
Jan 23, 2023
Merged

Use kvikIO as the default IO backend #12574

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
b75a4b7
change default
vuule Jan 18, 2023
87d7fb1
remove (now) redundant tests
vuule Jan 18, 2023
e425a7a
docs
vuule Jan 18, 2023
2e5f411
style
vuule Jan 19, 2023
d320823
Merge branch 'branch-23.02' of https://github.com/rapidsai/cudf into …
vuule Jan 20, 2023
fb5258f
Merge branch 'branch-23.02' into fea-kvikio-default
vuule Jan 23, 2023
fea159e
Merge branch 'branch-23.02' of https://github.com/rapidsai/cudf into …
vuule Jan 23, 2023
50bdf1f
test changes in new file
vuule Jan 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 0 additions & 14 deletions ci/gpu/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,13 +167,6 @@ if [[ -z "$PROJECT_FLASH" || "$PROJECT_FLASH" == "0" ]]; then
echo "Running GoogleTest $test_name"
${gt} --gtest_output=xml:"$WORKSPACE/test-results/"
done

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is no longer in use by CI since the switch to GitHub Actions.

the new CPP test file is here: https://github.com/rapidsai/cudf/blob/branch-23.02/ci/test_cpp.sh.

I imagine you may want to make similar changes there.

We will be removing all of the old Jenkins CI scripts from the RAPIDS repositories in the coming weeks.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, done.

# Test libcudf (csv, orc, and parquet) with `LIBCUDF_CUFILE_POLICY=KVIKIO`
for test_name in "CSV_TEST" "ORC_TEST" "PARQUET_TEST" "DATA_CHUNK_SOURCE_TEST"; do
gt="$WORKSPACE/cpp/build/gtests/$test_name"
echo "Running GoogleTest $test_name (LIBCUDF_CUFILE_POLICY=KVIKIO)"
LIBCUDF_CUFILE_POLICY=KVIKIO ${gt} --gtest_output=xml:"$WORKSPACE/test-results/"
done
fi
else
#Project Flash
Expand Down Expand Up @@ -231,13 +224,6 @@ else
fi
done

# Test libcudf (csv, orc, and parquet) with `LIBCUDF_CUFILE_POLICY=KVIKIO`
for test_name in "CSV_TEST" "ORC_TEST" "PARQUET_TEST" "DATA_CHUNK_SOURCE_TEST"; do
gt="$CONDA_PREFIX/bin/gtests/libcudf/$test_name"
echo "Running GoogleTest $test_name (LIBCUDF_CUFILE_POLICY=KVIKIO)"
LIBCUDF_CUFILE_POLICY=KVIKIO ${gt} --gtest_output=xml:"$WORKSPACE/test-results/"
done

export LIB_BUILD_DIR="$WORKSPACE/ci/artifacts/cudf/cpu/libcudf_work/cpp/build"
# Copy libcudf build time results
echo "Checking for build time log $LIB_BUILD_DIR/ninja_log.xml"
Expand Down
4 changes: 2 additions & 2 deletions cpp/src/io/utilities/config_utils.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2021-2022, NVIDIA CORPORATION.
* Copyright (c) 2021-2023, NVIDIA CORPORATION.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -42,7 +42,7 @@ enum class usage_policy : uint8_t { OFF, GDS, ALWAYS, KVIKIO };
*/
usage_policy get_env_policy()
{
static auto const env_val = getenv_or<std::string>("LIBCUDF_CUFILE_POLICY", "GDS");
static auto const env_val = getenv_or<std::string>("LIBCUDF_CUFILE_POLICY", "KVIKIO");
if (env_val == "OFF") return usage_policy::OFF;
if (env_val == "GDS") return usage_policy::GDS;
if (env_val == "ALWAYS") return usage_policy::ALWAYS;
Expand Down
2 changes: 1 addition & 1 deletion docs/cudf/source/user_guide/io/io.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ There are four valid values for the environment variable:
- "KVIKIO": Enable GDS through [KvikIO](https://github.com/rapidsai/kvikio).
- "OFF": Completely disable GDS use.

If no value is set, behavior will be the same as the "GDS" option.
If no value is set, behavior will be the same as the "KVIKIO" option.

This environment variable also affects how cuDF treats GDS errors.

Expand Down