BigQuery: Add BadGateway to DEFAULT_RETRY predicate #5918

bencaine1 · 2018-09-11T15:01:41Z

OS: Linux dc32b7e8763a 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02) x86_64 x86_64 x86_64 GNU/Linux
Python version: Python 2.7.6
google-cloud-bigquery: 1.5.0

I'm getting BadGateway exceptions periodically when running queries with the default retry object. Here's an example (relevant part of the stack trace):

  File "/opt/conda/lib/python2.7/site-packages/google/cloud/bigquery/job.py", line 2605, in done
    location=self.location)
  File "/opt/conda/lib/python2.7/site-packages/google/cloud/bigquery/client.py", line 579, in _get_query_results
    retry, method='GET', path=path, query_params=extra_params)
  File "/opt/conda/lib/python2.7/site-packages/google/cloud/bigquery/client.py", line 331, in _call_api
    return call()
  File "/opt/conda/lib/python2.7/site-packages/google/api_core/retry.py", line 260, in retry_wrapped_func
    on_error=on_error,
  File "/opt/conda/lib/python2.7/site-packages/google/api_core/retry.py", line 177, in retry_target
    return target()
  File "/opt/conda/lib/python2.7/site-packages/google/cloud/_http.py", line 293, in api_request
    raise exceptions.from_http_response(response)
BadGateway: 502 GET https://www.googleapis.com/bigquery/v2/projects/packard-campbell-synth/queries/c4af3c68-d204-455e-877f-9f81e1c77d68?maxResults=0: <!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 502 (Server Error)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>502.</b> <ins>That’s an error.</ins>
  <p>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.  <ins>That’s all we know.</ins>

It would be useful to add this to the DEFAULT_RETRY object in https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/bigquery/google/cloud/bigquery/retry.py.

The text was updated successfully, but these errors were encountered:

Closes #5918.

bencaine1 · 2018-09-13T13:19:39Z

I've patched the library at commit f3e00ad, and we're still getting BadGateways.

I think it's because the BadGateway is raised before the job is done, and therefore looking at the job's errors isn't going to show BadGateway. So I believe you'll need to catch BadGateway using something like the if_exception_type method (https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/api_core/google/api_core/retry.py#L78).

tseaver · 2018-09-13T18:12:53Z

@bencaine1 Thanks for the followup. The issue is that the BadGateway you report is being raised from Client._get_query_results, which doesn't take a retry argument.

tseaver · 2018-09-13T18:17:38Z

Oops, Client._get_query_results does take a retry argument, and QueryJob.result passes it.

tseaver · 2018-09-13T18:22:33Z

@bencaine1 Can you manage to trap the code when the BadGateway is raised, and print out the full response body?

@tswast Any clues from your side how we should be handling this?

tswast · 2018-09-13T19:18:38Z

It could be that errors or reason isn't getting populated? I think 502 comes from the layer before it actually hit's BigQuery, so maybe the BigQuery team's advice to look at the body rather than error codes shouldn't apply for 502?

tswast · 2018-09-13T19:18:51Z

CC @shollyman

tswast · 2018-09-13T19:24:11Z

From the error message, it looks like the body is HTML, not JSON, so probably the reason field isn't populated.

https://github.com/GoogleCloudPlatform/google-cloud-python/blob/64caa5f0a319313ccc5dbe515ae91c4029548447/api_core/google/api_core/exceptions.py#L362-L388

shollyman · 2018-09-13T20:29:59Z

Yes, this is likely a GFE response, which means we're not going to get a structured error back.

Closes #5918.

bencaine1 · 2018-09-27T19:53:58Z

Thanks! We've patched this version and we haven't seen any BadGateways yet. Will keep you posted :)

bencaine1 · 2018-10-04T14:23:45Z

Sorry to reopen a closed issue, but we just got another BadGateway.

Stack trace looks similar:

  File "/opt/conda/lib/python2.7/site-packages/google/cloud/bigquery/client.py", line 1232, in query
    query_job._begin(retry=retry)
  File "/opt/conda/lib/python2.7/site-packages/google/cloud/bigquery/job.py", line 543, in _begin
    method='POST', path=path, data=self._build_resource())
  File "/opt/conda/lib/python2.7/site-packages/google/cloud/bigquery/client.py", line 331, in _call_api
    return call()
  File "/opt/conda/lib/python2.7/site-packages/google/api_core/retry.py", line 260, in retry_wrapped_func
    on_error=on_error,
  File "/opt/conda/lib/python2.7/site-packages/google/api_core/retry.py", line 177, in retry_target
    return target()
  File "/opt/conda/lib/python2.7/site-packages/google/cloud/_http.py", line 293, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.BadGateway: 502 POST https://www.googleapis.com/bigquery/v2/projects/packard-campbell-synth/jobs: <!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 502 (Server Error)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>502.</b> <ins>That’s an error.</ins>
  <p>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.  <ins>That’s all we know.</ins>

Something I just noticed about this stack trace: it's failing on this line in api_core/retry.py:

  File "/opt/conda/lib/python2.7/site-packages/google/api_core/retry.py", line 177, in retry_target
    return target()

But that line is inside a try/except, where the except clause catches all Exceptions. So maybe this isn't even raising an Exception and going into the retry logic?

bencaine1 · 2018-10-22T19:11:51Z

@tseaver, PTAL when you get a chance. Thanks!

tseaver · 2018-10-22T20:35:54Z

@bencaine1 is your application passing a non-default retry argument to client.query?

tseaver added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. api: bigquery Issues related to the BigQuery API. priority: p2 Moderately-important priority. Fix may not be included in next release. labels Sep 11, 2018

tseaver mentioned this issue Sep 11, 2018

Retry '502 Bad Gateway' errors by default. #5930

Merged

tseaver closed this as completed in #5930 Sep 11, 2018

tseaver added a commit that referenced this issue Sep 11, 2018

Retry '502 Bad Gateway' errors by default. (#5930)

f3e00ad

Closes #5918.

tseaver reopened this Sep 13, 2018

tseaver added the backend label Sep 13, 2018

tseaver self-assigned this Sep 18, 2018

tseaver mentioned this issue Sep 18, 2018

BigQuery: add support for retrying unstructured 429 / 500 / 502 responses. #6011

Merged

tseaver closed this as completed in #6011 Sep 18, 2018

tseaver added a commit that referenced this issue Sep 18, 2018

Add support for retrying unstructured 429 / 500 / 502 responses. (#6011)

103ced7

Closes #5918.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BigQuery: Add BadGateway to DEFAULT_RETRY predicate #5918

BigQuery: Add BadGateway to DEFAULT_RETRY predicate #5918

bencaine1 commented Sep 11, 2018

bencaine1 commented Sep 13, 2018

tseaver commented Sep 13, 2018

tseaver commented Sep 13, 2018

tseaver commented Sep 13, 2018

tswast commented Sep 13, 2018

tswast commented Sep 13, 2018

tswast commented Sep 13, 2018

shollyman commented Sep 13, 2018

bencaine1 commented Sep 27, 2018

bencaine1 commented Oct 4, 2018 •

edited

Loading

bencaine1 commented Oct 22, 2018

tseaver commented Oct 22, 2018

BigQuery: Add BadGateway to DEFAULT_RETRY predicate #5918

BigQuery: Add BadGateway to DEFAULT_RETRY predicate #5918

Comments

bencaine1 commented Sep 11, 2018

bencaine1 commented Sep 13, 2018

tseaver commented Sep 13, 2018

tseaver commented Sep 13, 2018

tseaver commented Sep 13, 2018

tswast commented Sep 13, 2018

tswast commented Sep 13, 2018

tswast commented Sep 13, 2018

shollyman commented Sep 13, 2018

bencaine1 commented Sep 27, 2018

bencaine1 commented Oct 4, 2018 • edited Loading

bencaine1 commented Oct 22, 2018

tseaver commented Oct 22, 2018

bencaine1 commented Oct 4, 2018 •

edited

Loading