Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement exponential backoff for retrieving cloud build status #2667

Merged
merged 2 commits into from
Aug 22, 2019

Conversation

nkubala
Copy link
Contributor

@nkubala nkubala commented Aug 15, 2019

Previously, when several GCB builds were happening simultaneously (particularly long running builds), the rate limit would get hit for retrieving the build status, causing the build to fail. This implements an exponential backoff for retrieving this build status, so these jobs can keep running.

Fixes #1589

@codecov
Copy link

codecov bot commented Aug 15, 2019

Codecov Report

Merging #2667 into master will decrease coverage by 0.14%.
The diff coverage is 0%.

Impacted Files Coverage Δ
pkg/skaffold/build/gcb/cloud_build.go 0% <0%> (ø) ⬆️
pkg/skaffold/build/gcb/types.go 14.7% <0%> (-3.16%) ⬇️

Copy link
Contributor

@balopat balopat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with a minor modification

@balopat balopat merged commit e5a5cd3 into GoogleContainerTools:master Aug 22, 2019
@iquadrat
Copy link

Looks like the error message has changed and therefore the exponential backoff no longer works
ERROR: (gcloud.builds.submit) RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'Build and Operation Get requests' and limit 'Build and Operation Get requests per minute' of service 'cloudbuild.googleapis.com' for consumer
I am getting this error on gcloud builds submit when there are many builds pending.

@nkubala nkubala deleted the gcb-quota branch June 17, 2021 18:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

GCP cloud build quota
4 participants