Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove remaining Python 3.8 Artifacts #32913

Merged
merged 3 commits into from
Oct 29, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/workflows/update_python_dependencies.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,6 @@ jobs:
uses: ./.github/actions/setup-environment-action
with:
python-version: |
3.8
3.9
3.10
3.11
Expand Down
1 change: 1 addition & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@
## Deprecations

* Removed support for Flink 1.15 and 1.16
* Removed support for Python 3.8
* X behavior is deprecated and will be removed in X versions ([#X](https://github.com/apache/beam/issues/X)).

## Bugfixes
Expand Down
21 changes: 7 additions & 14 deletions build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -501,23 +501,11 @@ tasks.register("pythonFormatterPreCommit") {
dependsOn("sdks:python:test-suites:tox:pycommon:formatter")
}

tasks.register("python38PostCommit") {
dependsOn(":sdks:python:test-suites:dataflow:py38:postCommitIT")
dependsOn(":sdks:python:test-suites:direct:py38:postCommitIT")
dependsOn(":sdks:python:test-suites:direct:py38:hdfsIntegrationTest")
dependsOn(":sdks:python:test-suites:direct:py38:azureIntegrationTest")
dependsOn(":sdks:python:test-suites:portable:py38:postCommitPy38")
// TODO: https://github.com/apache/beam/issues/22651
// The default container uses Python 3.8. The goal here is to
// duild Docker images for TensorRT tests during run time for python versions
// other than 3.8 and add these tests in other python postcommit suites.
dependsOn(":sdks:python:test-suites:dataflow:py38:inferencePostCommitIT")
dependsOn(":sdks:python:test-suites:direct:py38:inferencePostCommitIT")
}

tasks.register("python39PostCommit") {
dependsOn(":sdks:python:test-suites:dataflow:py39:postCommitIT")
dependsOn(":sdks:python:test-suites:direct:py39:postCommitIT")
dependsOn(":sdks:python:test-suites:direct:py39:hdfsIntegrationTest")
dependsOn(":sdks:python:test-suites:direct:py39:azureIntegrationTest")
dependsOn(":sdks:python:test-suites:portable:py39:postCommitPy39")
// TODO (https://github.com/apache/beam/issues/23966)
// Move this to Python 3.10 test suite once tfx-bsl has python 3.10 wheel.
Expand All @@ -528,6 +516,11 @@ tasks.register("python310PostCommit") {
dependsOn(":sdks:python:test-suites:dataflow:py310:postCommitIT")
dependsOn(":sdks:python:test-suites:direct:py310:postCommitIT")
dependsOn(":sdks:python:test-suites:portable:py310:postCommitPy310")
// TODO: https://github.com/apache/beam/issues/22651
// The default container uses Python 3.10. The goal here is to
// duild Docker images for TensorRT tests during run time for python versions
// other than 3.10 and add these tests in other python postcommit suites.
dependsOn(":sdks:python:test-suites:dataflow:py310:inferencePostCommitIT")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this change broke the TensorRT tests - https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python.yml

The reason is that TensorRT depends on a docker image -

"sdk_container_image": "us.gcr.io/apache-beam-testing/python-postcommit-it/tensor_rt:latest",
- which is static, not built based on the test suite its in (this is actually what the comment in this file is saying).

I think we need to use

COPY --from=apache/beam_python3.10_sdk:latest /opt/apache/beam /opt/apache/beam
to build a new docker image for the tensorRT jobs and push it to us.gcr.io/apache-beam-testing/python-postcommit-it/tensor_rt:latest. @jrmccluskey would you mind following up and doing that? It looks like you already updated the dockerfile, so hopefully its smooth sailing

}

tasks.register("python311PostCommit") {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@

_LOGGER = logging.getLogger(__name__)

_PYTHON_VERSIONS_SUPPORTED_BY_DATAFLOW = ['3.8', '3.9', '3.10', '3.11', '3.12']
_PYTHON_VERSIONS_SUPPORTED_BY_DATAFLOW = ['3.9', '3.10', '3.11', '3.12']


class Environment(object):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1003,7 +1003,21 @@ def test_interpreter_version_check_passes_with_experiment(self):
'apache_beam.runners.dataflow.internal.apiclient.'
'beam_version.__version__',
'2.2.0')
def test_interpreter_version_check_passes_py38(self):
def test_interpreter_version_check_fails_py38(self):
pipeline_options = PipelineOptions([])
self.assertRaises(
Exception,
apiclient._verify_interpreter_version_is_supported,
pipeline_options)

@mock.patch(
'apache_beam.runners.dataflow.internal.apiclient.sys.version_info',
(3, 9, 6))
@mock.patch(
'apache_beam.runners.dataflow.internal.apiclient.'
'beam_version.__version__',
'2.2.0')
def test_interpreter_version_check_passes_py39(self):
pipeline_options = PipelineOptions([])
apiclient._verify_interpreter_version_is_supported(pipeline_options)

Expand Down
2 changes: 1 addition & 1 deletion sdks/python/container/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ plugins { id 'org.apache.beam.module' }
applyGoNature()

description = "Apache Beam :: SDKs :: Python :: Container"
int min_python_version=8
int min_python_version=9
int max_python_version=12

configurations {
Expand Down
172 changes: 0 additions & 172 deletions sdks/python/container/py38/base_image_requirements.txt

This file was deleted.

28 changes: 0 additions & 28 deletions sdks/python/container/py38/build.gradle

This file was deleted.

4 changes: 2 additions & 2 deletions sdks/python/test-suites/dataflow/common.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -543,8 +543,8 @@ task mockAPITests {
}

// add all RunInference E2E tests that run on DataflowRunner
// As of now, this test suite is enable in py38 suite as the base NVIDIA image used for Tensor RT
// contains Python 3.8.
// As of now, this test suite is enable in py310 suite as the base NVIDIA image used for Tensor RT
// contains Python 3.10.
// TODO: https://github.com/apache/beam/issues/22651
project.tasks.register("inferencePostCommitIT") {
dependsOn = [
Expand Down
24 changes: 0 additions & 24 deletions sdks/python/test-suites/dataflow/py38/build.gradle

This file was deleted.

24 changes: 0 additions & 24 deletions sdks/python/test-suites/direct/py38/build.gradle

This file was deleted.

26 changes: 0 additions & 26 deletions sdks/python/test-suites/portable/py38/build.gradle

This file was deleted.

Loading
Loading