Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to using native traceback #16851

Merged
merged 6 commits into from
Sep 20, 2024
Merged

Switch to using native traceback #16851

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
8ee78ae
Switch to using native traceback
galipremsagar Sep 19, 2024
bce16a9
Merge branch 'branch-24.10' into tb_optimize
galipremsagar Sep 19, 2024
bca0a1c
Merge remote-tracking branch 'upstream/branch-24.10' into tb_optimize
galipremsagar Sep 19, 2024
573cf49
Merge branch 'tb_optimize' of https://github.com/galipremsagar/cudf i…
galipremsagar Sep 19, 2024
ed5b0e4
Merge remote-tracking branch 'upstream/branch-24.10' into tb_optimize
galipremsagar Sep 19, 2024
52a29fc
move to pytest.ini
galipremsagar Sep 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions ci/test_python_cudf.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ pushd python/pylibcudf/pylibcudf/tests
python -m pytest \
--cache-clear \
--dist=worksteal \
--tb=native \
.
popd

Expand All @@ -30,7 +31,8 @@ rapids-logger "pytest cudf"
--cov-config=../.coveragerc \
--cov=cudf \
--cov-report=xml:"${RAPIDS_COVERAGE_DIR}/cudf-coverage.xml" \
--cov-report=term
--cov-report=term \
--tb=native

# Run benchmarks with both cudf and pandas to ensure compatibility is maintained.
# Benchmarks are run in DEBUG_ONLY mode, meaning that only small data sizes are used.
Expand All @@ -44,7 +46,8 @@ rapids-logger "pytest for cudf benchmarks"
--cov-config=.coveragerc \
--cov=cudf \
--cov-report=xml:"${RAPIDS_COVERAGE_DIR}/cudf-benchmark-coverage.xml" \
--cov-report=term
--cov-report=term \
--tb=native

rapids-logger "pytest for cudf benchmarks using pandas"
./ci/run_cudf_pandas_pytest_benchmarks.sh \
Expand All @@ -53,7 +56,8 @@ rapids-logger "pytest for cudf benchmarks using pandas"
--cov-config=.coveragerc \
--cov=cudf \
--cov-report=xml:"${RAPIDS_COVERAGE_DIR}/cudf-benchmark-pandas-coverage.xml" \
--cov-report=term
--cov-report=term \
--tb=native

rapids-logger "Test script exiting with value: $EXITCODE"
exit ${EXITCODE}
10 changes: 7 additions & 3 deletions ci/test_python_other.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,18 +27,21 @@ DASK_DATAFRAME__QUERY_PLANNING=True ./ci/run_dask_cudf_pytests.sh \
--cov-config=../.coveragerc \
--cov=dask_cudf \
--cov-report=xml:"${RAPIDS_COVERAGE_DIR}/dask-cudf-coverage.xml" \
--cov-report=term
--cov-report=term \
--tb=native

rapids-logger "pytest dask_cudf (legacy)"
DASK_DATAFRAME__QUERY_PLANNING=False ./ci/run_dask_cudf_pytests.sh \
--junitxml="${RAPIDS_TESTS_DIR}/junit-dask-cudf-legacy.xml" \
--numprocesses=8 \
--dist=loadscope \
--tb=native \
.

rapids-logger "pytest cudf_kafka"
./ci/run_cudf_kafka_pytests.sh \
--junitxml="${RAPIDS_TESTS_DIR}/junit-cudf-kafka.xml"
--junitxml="${RAPIDS_TESTS_DIR}/junit-cudf-kafka.xml" \
--tb=native

rapids-logger "pytest custreamz"
./ci/run_custreamz_pytests.sh \
Expand All @@ -48,7 +51,8 @@ rapids-logger "pytest custreamz"
--cov-config=../.coveragerc \
--cov=custreamz \
--cov-report=xml:"${RAPIDS_COVERAGE_DIR}/custreamz-coverage.xml" \
--cov-report=term
--cov-report=term \
--tb=native

rapids-logger "Test script exiting with value: $EXITCODE"
exit ${EXITCODE}
3 changes: 3 additions & 0 deletions ci/test_wheel_cudf.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,9 @@ rapids-logger "pytest pylibcudf"
pushd python/pylibcudf/pylibcudf/tests
python -m pytest \
--cache-clear \
--numprocesses=8 \
--dist=worksteal \
--tb=native
.
popd

Expand All @@ -50,5 +52,6 @@ python -m pytest \
--junitxml="${RAPIDS_TESTS_DIR}/junit-cudf.xml" \
--numprocesses=8 \
--dist=worksteal \
--tb=native
.
popd
3 changes: 2 additions & 1 deletion ci/test_wheel_cudf_polars.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,8 @@ set +e
--cov cudf_polars \
--cov-fail-under=100 \
--cov-config=./pyproject.toml \
--junitxml="${RAPIDS_TESTS_DIR}/junit-cudf-polars.xml"
--junitxml="${RAPIDS_TESTS_DIR}/junit-cudf-polars.xml" \
--tb=native

trap ERR
set -e
Expand Down
4 changes: 4 additions & 0 deletions ci/test_wheel_dask_cudf.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ pushd python/dask_cudf/dask_cudf
DASK_DATAFRAME__QUERY_PLANNING=True python -m pytest \
--junitxml="${RAPIDS_TESTS_DIR}/junit-dask-cudf.xml" \
--numprocesses=8 \
--dist=worksteal \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious how much of a speedup you see with this change?

Copy link
Contributor Author

@galipremsagar galipremsagar Sep 20, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we switched cudf wide, we were able to cut down 25% of the execution time: #15207 (comment)

--tb=native \
.
popd

Expand All @@ -50,5 +52,7 @@ pushd python/dask_cudf/dask_cudf
DASK_DATAFRAME__QUERY_PLANNING=False python -m pytest \
--junitxml="${RAPIDS_TESTS_DIR}/junit-dask-cudf-legacy.xml" \
--numprocesses=8 \
--dist=worksteal \
--tb=native \
.
popd
Loading