Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace StorageV1 client with GCS client (take 2) #27471

Closed
wants to merge 85 commits into from

Conversation

BjornPrime
Copy link
Contributor

Addresses #25676.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI.

@BjornPrime BjornPrime changed the title Replace StorageV1 client with GCS client Replace StorageV1 client with GCS client (take 2) Jul 12, 2023
@BjornPrime
Copy link
Contributor Author

Run Python 3.11 PostCommit

@github-actions
Copy link
Contributor

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @liferoad for label python.
R: @manavgarg for label io.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@BjornPrime
Copy link
Contributor Author

Run Python 3.11 PostCommit

zechenj18 and others added 10 commits August 7, 2023 10:51
* Fix broken url for 'Tour of Beam'

* Update examples/notebooks/get-started/learn_beam_basics_by_doing.ipynb

Co-authored-by: Danny McCormick <[email protected]>

* Fix unrelated diff

---------

Co-authored-by: Danny McCormick <[email protected]>
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.10.0 to 0.11.0.
- [Commits](golang/oauth2@v0.10.0...v0.11.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
author Bjorn Pedersen <[email protected]> 1688152191 -0400
committer Bjorn Pedersen <[email protected]> 1691418620 -0400

Replace StorageV1 client with GCS client

first attempt at standardizing responses

tinkering to get a better error message

trying out appending the response tuples

defining success response codes more broadly

fixed re-delete behavior

added assertions checking that request and response lengths are equal

changed delete_blobs to delete_blob

fix apiclient bug

making juliaset work size smaller to see if it is actually timing out

apiclient debugging

removed test_user_agent_insert_all

tinkering with apiclient

replacing return value

commenting out set-up file

restoring setup file to see if that induces test failure

Revert "restoring setup file to see if that induces test failure"

This reverts commit c0af90c.

Revert "Revert "restoring setup file to see if that induces test failure""

This reverts commit a339f4c.

enabled raw_download for uncompressed files

fixing unit tests

set raw_download to always be true

commented out setup_file again

Revert "commented out setup_file again"

This reverts commit 157311e.

reverting raw_download implementation

gcs filesystem no longer forces mime_type to match compression

updated stage_file to use BlobWriter directly, rather than go through FileSystem

corrected BlobWriter import

restoring gcsio batch response work

Revert "making juliaset work size smaller to see if it is actually timing out"

This reverts commit 582b162.

Revert "gcs filesystem no longer forces mime_type to match compression"

This reverts commit c04ed94.
@BjornPrime
Copy link
Contributor Author

Ran into some issues with rogue commits to this branch. Updated version (using new branch) at #27878.

@BjornPrime BjornPrime closed this Aug 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.