BigQuery: Add "Forbidden: 403 Exceeded rate limits" to list of retryable exceptions

[bencaine1]

OS: Linux dc32b7e8763a 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02) x86_64 x86_64 x86_64 GNU/Linux
Python version: Python 2.7.6
google-cloud-bigquery: 1.6.0

We're getting the following error:

Forbidden: 403 Exceeded rate limits: too many concurrent queries for this project. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors

In _should_retry, it should retry if isinstance(exc, Forbidden) and 'Exceeded rate limits' in str(exc).

[tseaver]

@bencaine1 Thanks for the report!

@tswast Can you please figure out with the back-end team why they would return a 403 here, rather than the expected 429? RFC 2616 specifically states about a 403:

The server understood the request, but is refusing to fulfill it. Authorization will not help and the request SHOULD NOT be repeated.

Versus what it RFC 6585 for 429:

The 429 status code indicates that the user has sent too many requests in a given amount of time ("rate limiting").
The response representations SHOULD include details explaining the condition, and MAY include a Retry-After header indicating how long to wait before making a new request.

[shollyman]

The short answer for why ratelimit pushback is communicated back using 403 vs 429 is that the behavior
in BigQuery predates the updated RFC6585. I've filed an internal issue (119203044) with the team to consider modifications to this behavior.

[tswast]

FYI: internal issue 119203044 was closed as wontfix, as this would be a breaking change. The client-side workaround would be to look for "Exceeded rate limits" and retry in that case. (Also, it might actually make sense to always retry on 403 to give the auth library a chance to refresh credentials.)

[yan-hic]

@twast any traction on this ? We have implemented our own resubmit logic for loads but we are now hitting a lot of quota limits on queries too.
Looks like job.result() could be the place to resubmit (and change the jobId).

[tswast]

Since this requires retrying multiple API requests (not just the "get query results" request that is actually raising the error), this feature request will require careful design work before we can proceed with implementation.

CC @crwilcox

[tswast]

Closing in favor of https://github.com/googleapis/google-cloud-python/issues/9781