GH-38323: [CI][Python] Use system gdb on test-conda-python #38324

pitrou · 2023-10-18T11:52:17Z

Rationale for this change

gdb on conda-forge frequently conflicts with the newest Python version (example for Python 3.12: conda-forge/gdb-feedstock#66)

What changes are included in this PR?

Use the system-provided gdb instead of trying to install it from conda-forge.

This works because the Python interpreter used for gdb is entirely independent from the Python interpreter under test, and gdb itself can debug any binary, regardless of binutils version.

However, there is a complication: if the system Python version and the conda-forge Python version are the same, then for some reason pyarrow/tests/test_gdb.py would fail with the system gdb. In that case we still install the conda-forge gdb.

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

Closes: [CI][Python] Image build failure on test-conda-python-3.12 #38323

gdb on conda-forge frequently conflicts with the newest Python version (example for Python 3.12: conda-forge/gdb-feedstock#66) We needn't install gdb from conda-forge, though, as the Python interpreter used for gdb is entirely independent from the Python interpreter under test.

pitrou · 2023-10-18T11:52:46Z

@github-actions crossbow submit test-conda-python*

pitrou · 2023-10-18T12:29:40Z

Hmm, things are not so simple. All builds pass except when the Python version under test is the same as the one that the system gdb is linked against, in which case the tests fail.

…version

github-actions · 2023-10-18T12:46:06Z

Revision: cc0ad1f

Submitted crossbow builds: ursacomputing/crossbow @ actions-add7835c90

Task	Status
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest
test-conda-python-3.10-pandas-nightly
test-conda-python-3.10-spark-v3.5.0
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-upstream_devel
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.8
test-conda-python-3.8-pandas-1.0
test-conda-python-3.8-spark-v3.5.0
test-conda-python-3.9
test-conda-python-3.9-pandas-latest

pitrou · 2023-10-18T13:09:41Z

ci/docker/conda-python.dockerfile

+# If the Python version being tested is the same as the Python used by the system gdb,
+# we need to install the conda-forge gdb instead.
 RUN mamba install -q -y \
        --file arrow/ci/conda_env_python.txt \
-        gdb \
+        $([ "$python" == "3.10" ] && echo "gdb") \


@kou Does this look ok?

Will this be broken when we change the base Ubuntu version?

arrow/ci/docker/conda.dockerfile

Line 19 in 63840f2

FROM ${arch}/ubuntu:22.04

Can we detect the Python version that is used by system GDB?
gdb --batch --eval-command 'python import sys; print(sys.version_info.major, sys.version_info.minor, sep=".")'?

Hmm, I'm not sure how to do that in a Dockerfile. Can you help me?

kou

All builds pass except when the Python version under test is the same as the one that the system gdb is linked against, in which case the tests fail.

Do you know why?

Could you update the description before we merge this?

kou · 2023-10-19T02:22:04Z

ci/docker/conda-python.dockerfile

+# If the Python version being tested is the same as the Python used by the system gdb,
+# we need to install the conda-forge gdb instead.
 RUN mamba install -q -y \
        --file arrow/ci/conda_env_python.txt \
-        gdb \
+        $([ "$python" == "3.10" ] && echo "gdb") \


Will this be broken when we change the base Ubuntu version?

arrow/ci/docker/conda.dockerfile

Line 19 in 63840f2

FROM ${arch}/ubuntu:22.04

Can we detect the Python version that is used by system GDB?
gdb --batch --eval-command 'python import sys; print(sys.version_info.major, sys.version_info.minor, sep=".")'?

pitrou · 2023-10-19T07:22:18Z

Do you know why?

I honestly don't understand why. But it only happens when testing Python 3.10 (which is also the Python version installed under Ubuntu)...

kou · 2023-10-19T07:44:51Z

ci/docker/conda-python.dockerfile

 RUN mamba install -q -y \
        --file arrow/ci/conda_env_python.txt \
-        gdb \
+        $([ "$python" == "3.10" ] && echo "gdb") \


Can we try this?

Suggested change

$([ "$python" == "3.10" ] && echo "gdb") \

$([ "$python" = $(gdb --batch --eval-command 'python import sys; print(sys.version_info.major, sys.version_info.minor, sep=".")') ] && echo "gdb") \

pitrou · 2023-10-19T08:02:19Z

@github-actions crossbow submit test-conda-python*

github-actions · 2023-10-19T08:05:34Z

Revision: 53a1887

Submitted crossbow builds: ursacomputing/crossbow @ actions-8f00837e89

Task	Status
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest
test-conda-python-3.10-pandas-nightly
test-conda-python-3.10-spark-v3.5.0
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-upstream_devel
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.8
test-conda-python-3.8-pandas-1.0
test-conda-python-3.8-spark-v3.5.0
test-conda-python-3.9
test-conda-python-3.9-pandas-latest

conbench-apache-arrow · 2023-10-21T22:56:02Z

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 883a439.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them.

…che#38324) ### Rationale for this change gdb on conda-forge frequently conflicts with the newest Python version (example for Python 3.12: conda-forge/gdb-feedstock#66) ### What changes are included in this PR? Use the system-provided gdb instead of trying to install it from conda-forge. This works because the Python interpreter used for gdb is entirely independent from the Python interpreter under test, and gdb itself can debug any binary, regardless of binutils version. However, there is a complication: if the system Python version and the conda-forge Python version are the same, then for some reason `pyarrow/tests/test_gdb.py` would fail with the system gdb. In that case we still install the conda-forge gdb. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: apache#38323 Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

pitrou requested review from assignUser, kou and raulcd as code owners October 18, 2023 11:52

pitrou removed request for kou and assignUser October 18, 2023 11:52

github-actions bot added the awaiting review Awaiting review label Oct 18, 2023

This comment was marked as outdated.

Sign in to view

Still install gdb when testing the same Python version as the system …

cc0ad1f

…version

This comment was marked as outdated.

Sign in to view

pitrou commented Oct 18, 2023

View reviewed changes

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Oct 18, 2023

kou reviewed Oct 19, 2023

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Oct 19, 2023

kou reviewed Oct 19, 2023

View reviewed changes

Apply suggestion

53a1887

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 19, 2023

pitrou merged commit 883a439 into apache:main Oct 19, 2023

pitrou removed the awaiting change review Awaiting change review label Oct 19, 2023

pitrou deleted the gh38323-conda-system-gdb branch October 19, 2023 09:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-38323: [CI][Python] Use system gdb on test-conda-python #38324

GH-38323: [CI][Python] Use system gdb on test-conda-python #38324

pitrou commented Oct 18, 2023 •

edited

Loading

pitrou commented Oct 18, 2023

This comment was marked as outdated.

pitrou commented Oct 18, 2023

This comment was marked as outdated.

github-actions bot commented Oct 18, 2023

pitrou Oct 18, 2023

kou Oct 19, 2023

pitrou Oct 19, 2023

kou left a comment

kou Oct 19, 2023

pitrou commented Oct 19, 2023 •

edited

Loading

kou Oct 19, 2023

pitrou commented Oct 19, 2023

github-actions bot commented Oct 19, 2023

conbench-apache-arrow bot commented Oct 21, 2023

	$([ "$python" == "3.10" ] && echo "gdb") \
	$([ "$python" = $(gdb --batch --eval-command 'python import sys; print(sys.version_info.major, sys.version_info.minor, sep=".")') ] && echo "gdb") \

GH-38323: [CI][Python] Use system gdb on test-conda-python #38324

GH-38323: [CI][Python] Use system gdb on test-conda-python #38324

Conversation

pitrou commented Oct 18, 2023 • edited Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

pitrou commented Oct 18, 2023

This comment was marked as outdated.

pitrou commented Oct 18, 2023

This comment was marked as outdated.

github-actions bot commented Oct 18, 2023

pitrou Oct 18, 2023

Choose a reason for hiding this comment

kou Oct 19, 2023

Choose a reason for hiding this comment

pitrou Oct 19, 2023

Choose a reason for hiding this comment

kou left a comment

Choose a reason for hiding this comment

kou Oct 19, 2023

Choose a reason for hiding this comment

pitrou commented Oct 19, 2023 • edited Loading

kou Oct 19, 2023

Choose a reason for hiding this comment

pitrou commented Oct 19, 2023

github-actions bot commented Oct 19, 2023

conbench-apache-arrow bot commented Oct 21, 2023

pitrou commented Oct 18, 2023 •

edited

Loading

pitrou commented Oct 19, 2023 •

edited

Loading