Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cache GCS client #245

Merged
merged 7 commits into from
Dec 19, 2024
Merged

Cache GCS client #245

merged 7 commits into from
Dec 19, 2024

Conversation

2015aroras
Copy link
Contributor

A new GCS client is created each time a cached_path call is made to GCS. Making too many of them seems to cause Compute Engine Metadata server unavailable error and timeouts. Thus, this change caches the client.

@2015aroras 2015aroras requested a review from epwalsh December 19, 2024 00:48
epwalsh
epwalsh approved these changes Dec 19, 2024
View reviewed changes
Copy link
Member

@epwalsh epwalsh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we do something similar for S3? Other than the CI issue, LGTM

@2015aroras
Copy link
Contributor Author

b45bec3 Done for R2 and S3

c89e350 functools.cache is only available from python 3.9 onwards. Python 3.8 is end-of-life, so I've opted to just remove its workflow

@epwalsh
Copy link
Member

epwalsh commented Dec 19, 2024

Fine with me!

@2015aroras 2015aroras merged commit 1d209b8 into main Dec 19, 2024
9 of 10 checks passed
@2015aroras 2015aroras deleted the shanea/cache-gs-client branch December 19, 2024 19:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@epwalsh epwalsh epwalsh approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@2015aroras @epwalsh