Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Core: add retry on 503 errors. #5410

Closed
bmenasha opened this issue May 30, 2018 · 3 comments · Fixed by #8182
Closed

Core: add retry on 503 errors. #5410

bmenasha opened this issue May 30, 2018 · 3 comments · Fixed by #8182
Assignees
Labels
api: core type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@bmenasha
Copy link

bmenasha commented May 30, 2018

The google-api-python-client and gcloud CLI both retry on 503 SERVICE UNAVAILABLE errors yet the current version of google-cloud-python doesn't.

https://github.com/google/google-api-python-client/blob/master/googleapiclient/http.py#L94

We seem to have coded a retry on 500 and 429 , but not 503.

if_transient_error = if_exception_type((
    exceptions.InternalServerError,
    exceptions.TooManyRequests))

Please consider adding 503 as retryable.

I'm running 2.7.11 but I expect same behavior with 3

503 The service is currently unavailable.
  File "./create_policies.py", line 48, in <module>
    <class 'google.api_core.exceptions.ServiceUnavailable'>
print('Created ' +  future.result().name)
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/concurrent/futures/_base.py", line 462, in result
    return self.__get_result()
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/concurrent/futures/thread.py", line 63, in run
    result = self.fn(*self.args, **self.kwargs)
  File "./create_policies.py", line 11, in create_policy
    policy = alert_client.create_alert_policy(project, policy)
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/cloud/monitoring_v3/gapic/alert_policy_service_client.py", line 399, in create_alert_policy
    request, retry=retry, timeout=timeout, metadata=metadata)
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/api_core/gapic_v1/method.py", line 139, in __call__
    return wrapped_func(*args, **kwargs)
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/api_core/retry.py", line 263, in retry_wrapped_func
    on_error=on_error,
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/api_core/retry.py", line 178, in retry_target
    return target()
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/api_core/timeout.py", line 206, in func_with_timeout
    return func(*args, **kwargs)
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/api_core/grpc_helpers.py", line 56, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/six.py", line 737, in raise_from
    raise value
google.api_core.exceptions.ServiceUnavailable: 503 The service is currently unavailable.
---
Metadata-Version: 2.0
Name: google-cloud
Version: 0.22.0
Summary: API Client library for Google Cloud
Home-page: https://github.com/GoogleCloudPlatform/google-cloud-python
Author: Google Cloud Platform
Author-email: [email protected]
License: Apache 2.0
Location: /home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages
Requires: google-cloud-resource-manager, google-cloud-monitoring, google-cloud-happybase, google-cloud-language, google-cloud-storage, google-cloud-translate, google-cloud-error-reporting, google-cloud-vision, google-cloud-runtimeconfig, google-cloud-core, google-cloud-bigtable, google-cloud-bigquery, google-cloud-pubsub, google-cloud-datastore, google-cloud-logging, google-cloud-dns
@tseaver tseaver added type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. api: core labels May 31, 2018
@tseaver
Copy link
Contributor

tseaver commented May 31, 2018

@bmenasha Thanks for the report! @theacodes retrying 503 responses seems reasonable to me.

One quick note: the google-cloud umbrella package is deprecated, and will stop including any of the API-specific packages as dependencies in its next release (about three weeks from now). After that point, you will need to install the API-specific packages directly.

@xmedeko
Copy link

xmedeko commented May 27, 2019

A storage copy operation fenced by Retry just failed for me by 503, too. Also, the polling.RETRY_PREDICATE contains BadGateway HTTP 502 exception.

Please, fix this simple issue, add ServiceUnavailable and BadGateway to if_transient_error and uniry it with polling.RETRY_PREDICATE.

Ugly hack until the issue is solved:

orig_if_transient_error = retry.if_transient_error
retry.if_transient_error = lambda ex: orig_if_transient_error(ex) \
                                      or isinstance(ex, (exceptions.ServiceUnavailable, exceptions.BadGateway))
retry.Retry.__init__ = functools.partialmethod(retry.Retry.__init__, predicate = retry.if_transient_error)

@tseaver tseaver changed the title Should retry on 503 errors (Core) Core: add retry on 503 errors. May 28, 2019
tseaver added a commit that referenced this issue May 28, 2019
@xmedeko
Copy link

xmedeko commented May 28, 2019

@tseaver Thanks for the fix. What about BadGateway HTTP 502 exception? It's contained in the polling.RETRY_PREDICATE too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: core type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants