Update dataflow container tag #29790

AnandInguva · 2023-12-16T00:27:19Z

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

AnandInguva · 2023-12-16T00:27:58Z

R : @damccorm @tvalentyn @jrmccluskey

damondouglas

LGTM

github-actions · 2023-12-16T00:49:26Z

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

tvalentyn · 2023-12-18T11:32:05Z

FYI the image might not have the changes you need:

docker run --rm -it --entrypoint=/bin/bash gcr.io/cloud-dataflow/v1beta3/beam_python3.11_sdk:beam-master-20231215
Status: Downloaded newer image for gcr.io/cloud-dataflow/v1beta3/beam_python3.11_sdk:beam-master-20231215
root@c33b833dc7c7:/# pip list | grep storage
google-cloud-bigquery-storage 2.23.0
google-cloud-storage 2.13.0

tvalentyn · 2023-12-18T11:33:10Z

i'll re-run internal cloning workflow to clone latest images with the same tag.

tvalentyn · 2023-12-18T14:21:07Z

Actually, container deps still require google-cloud-storage==2.13.0. I assume you are making this PR to pick up 2.14.0.

beam/sdks/python/container/py310/base_image_requirements.txt

Line 67 in 539f042

google-cloud-storage==2.13.0

Also, I am getting a weird error when I try to install 2.14.0 manually in latest container, hopefully something is off just on my machine

root@229c00650217:/# pip install google-cloud-storage==2.14.0

Collecting google-cloud-storage==2.14.0
  Downloading google_cloud_storage-2.14.0-py2.py3-none-any.whl.metadata (6.1 kB)
Requirement already satisfied: google-auth<3.0dev,>=2.23.3 in /usr/local/lib/python3.10/site-packages (from google-cloud-storage==2.14.0) (2.25.1)
Requirement already satisfied: google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5 in /usr/local/lib/python3.10/site-packages (from google-cloud-storage==2.14.0) (2.14.0)
Requirement already satisfied: google-cloud-core<3.0dev,>=2.3.0 in /usr/local/lib/python3.10/site-packages (from google-cloud-storage==2.14.0) (2.3.3)
Requirement already satisfied: google-resumable-media>=2.6.0 in /usr/local/lib/python3.10/site-packages (from google-cloud-storage==2.14.0) (2.6.0)
Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /usr/local/lib/python3.10/site-packages (from google-cloud-storage==2.14.0) (2.31.0)
Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /usr/local/lib/python3.10/site-packages (from google-cloud-storage==2.14.0) (1.5.0)
Requirement already satisfied: googleapis-common-protos<2.0.dev0,>=1.56.2 in /usr/local/lib/python3.10/site-packages (from google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-storage==2.14.0) (1.61.0)
Requirement already satisfied: protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0.dev0,>=3.19.5 in /usr/local/lib/python3.10/site-packages (from google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-storage==2.14.0) (4.25.1)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/site-packages (from google-auth<3.0dev,>=2.23.3->google-cloud-storage==2.14.0) (5.3.2)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/site-packages (from google-auth<3.0dev,>=2.23.3->google-cloud-storage==2.14.0) (0.3.0)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/site-packages (from google-auth<3.0dev,>=2.23.3->google-cloud-storage==2.14.0) (4.9)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/site-packages (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.14.0) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/site-packages (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.14.0) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/site-packages (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.14.0) (2.1.0)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/site-packages (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.14.0) (2023.11.17)
Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/site-packages (from pyasn1-modules>=0.2.1->google-auth<3.0dev,>=2.23.3->google-cloud-storage==2.14.0) (0.5.1)
Downloading google_cloud_storage-2.14.0-py2.py3-none-any.whl (121 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/121.6 kB ? eta -:--:--ERROR: Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
    return func(self, options, args)
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 377, in run
    requirement_set = resolver.resolve(
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 179, in resolve
    self.factory.preparer.prepare_linked_requirements_more(reqs)
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 552, in prepare_linked_requirements_more
    self._complete_partial_requirements(
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 467, in _complete_partial_requirements
    for link, (filepath, _) in batch_download:
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/network/download.py", line 183, in __call__
    for chunk in chunks:
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/cli/progress_bars.py", line 52, in _rich_progress_bar
    with progress:
  File "/usr/local/lib/python3.10/site-packages/pip/_vendor/rich/progress.py", line 1169, in __enter__
    self.start()
  File "/usr/local/lib/python3.10/site-packages/pip/_vendor/rich/progress.py", line 1160, in start
    self.live.start(refresh=True)
  File "/usr/local/lib/python3.10/site-packages/pip/_vendor/rich/live.py", line 132, in start
    self._refresh_thread.start()
  File "/usr/local/lib/python3.10/threading.py", line 935, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

AnandInguva · 2023-12-18T15:23:13Z

The change I need was google-cloud-bigtable==2.22.0 and google-cloud-storage>=2.14.0. Let me check the container.

AnandInguva · 2023-12-18T15:56:23Z

google-cloud-aiplatform         1.37.0
google-cloud-bigquery           3.13.0
google-cloud-bigquery-storage   2.23.0
google-cloud-bigtable           2.22.0
google-cloud-core               2.3.3
google-cloud-datastore          2.18.0
google-cloud-dlp                3.13.0
google-cloud-language           2.11.1
google-cloud-profiler           4.1.0
google-cloud-pubsub             2.18.4
google-cloud-pubsublite         1.8.3
google-cloud-recommendations-ai 0.10.5
google-cloud-resource-manager   1.10.4
google-cloud-spanner            3.40.1
google-cloud-storage            2.13.0
google-cloud-videointelligence  2.11.4
google-cloud-vision             3.4.5

These are the google cloud dependencies in the container.

tvalentyn · 2023-12-18T16:03:15Z

fyi, there as a recent change: https://github.com/apache/beam/pull/29765/files#diff-1275c48808de339ef6f282d844c83ec441b5cfa0debc373fdcb7dba497da4fc8R376

tvalentyn · 2023-12-18T16:03:51Z

google-cloud-storage>=2.14.0 is missing

AnandInguva · 2023-12-18T16:07:35Z

Yes for the commit 1ce707c, the base image requirements was not updated. So the cloning containers is not picking up the change. I will make a PR updating base image requirements and then merge the new container tag after we create a second clone with same tag.

AnandInguva · 2023-12-18T16:16:48Z

#29800 - this will update the python base image dependencies.

AnandInguva · 2023-12-19T07:34:58Z

I reran the cloning containers. latest container dependencies

google-cloud-aiplatform         1.38.1
google-cloud-bigquery           3.14.1
google-cloud-bigquery-storage   2.24.0
google-cloud-bigtable           2.22.0
google-cloud-core               2.4.1
google-cloud-datastore          2.19.0
google-cloud-dlp                3.14.0
google-cloud-language           2.12.0
google-cloud-profiler           4.1.0
google-cloud-pubsub             2.19.0
google-cloud-pubsublite         1.9.0
google-cloud-recommendations-ai 0.10.6
google-cloud-resource-manager   1.11.0
google-cloud-spanner            3.40.1
google-cloud-storage            2.14.0
google-cloud-videointelligence  2.12.0
google-cloud-vision             3.5.0

Update container tag

718d20a

github-actions bot added the python label Dec 16, 2023

damondouglas approved these changes Dec 16, 2023

View reviewed changes

tvalentyn merged commit 87794aa into master Dec 19, 2023
67 of 71 checks passed

tvalentyn deleted the update_tag branch December 19, 2023 10:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update dataflow container tag #29790

Update dataflow container tag #29790

AnandInguva commented Dec 16, 2023

AnandInguva commented Dec 16, 2023

damondouglas left a comment

github-actions bot commented Dec 16, 2023

tvalentyn commented Dec 18, 2023

tvalentyn commented Dec 18, 2023

tvalentyn commented Dec 18, 2023

AnandInguva commented Dec 18, 2023 •

edited

Loading

AnandInguva commented Dec 18, 2023 •

edited

Loading

tvalentyn commented Dec 18, 2023

tvalentyn commented Dec 18, 2023

AnandInguva commented Dec 18, 2023 •

edited

Loading

AnandInguva commented Dec 18, 2023

AnandInguva commented Dec 19, 2023

Update dataflow container tag #29790

Update dataflow container tag #29790

Conversation

AnandInguva commented Dec 16, 2023

GitHub Actions Tests Status (on master branch)

AnandInguva commented Dec 16, 2023

damondouglas left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 16, 2023

tvalentyn commented Dec 18, 2023

tvalentyn commented Dec 18, 2023

tvalentyn commented Dec 18, 2023

AnandInguva commented Dec 18, 2023 • edited Loading

AnandInguva commented Dec 18, 2023 • edited Loading

tvalentyn commented Dec 18, 2023

tvalentyn commented Dec 18, 2023

AnandInguva commented Dec 18, 2023 • edited Loading

AnandInguva commented Dec 18, 2023

AnandInguva commented Dec 19, 2023

AnandInguva commented Dec 18, 2023 •

edited

Loading

AnandInguva commented Dec 18, 2023 •

edited

Loading

AnandInguva commented Dec 18, 2023 •

edited

Loading