Bigquery: it should retry on BigQueryException: 403 Forbidden Exceeded rate limits

[jneira-stratio]

Environment details

1. OS type and version: linux (does not matter)
2. Java version: 1.8
3. version(s): 2.11.2 (But behaviour would be the same in newer versions afaiu, nos tested though)

Steps to reproduce

1. Make server return this response with status 403:

{
  "code" : 403,
  "errors" : [ {
    "domain" : "usageLimits",
    "message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
    "reason" : "rateLimitExceeded"
  } ],
  "message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
  "status" : "PERMISSION_DENIED"
}

2. Observe there are no retries

This have been tested mocking the server with wiremock and returning exactly that response, verifying the trace produced matched the production environment one

Code example

// example
bigQueryService.listDatasets(
    bigQueryConnection.getProjectId,
    DatasetListOption.pageSize(Configuration.bigQueryDatasetPageSize)
)

Stack trace

Caused by: com.google.cloud.bigquery.BigQueryException: Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.getTable(HttpBigQueryRpc.java:299)
	at com.google.cloud.bigquery.BigQueryImpl$18.call(BigQueryImpl.java:778)
	at com.google.cloud.bigquery.BigQueryImpl$18.call(BigQueryImpl.java:775)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	at com.google.cloud.bigquery.BigQueryImpl.getTable(BigQueryImpl.java:774)
	at .....
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
GET https://www.googleapis.com/bigquery/v2/projects/myproject/datasets/mydataset/tables/mytable?prettyPrint=false
{
  "code" : 403,
  "errors" : [ {
    "domain" : "usageLimits",
    "message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
    "reason" : "rateLimitExceeded"
  } ],
  "message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
  "status" : "PERMISSION_DENIED"
}
	at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.getTable(HttpBigQueryRpc.java:297)
	... 44 common frames omitted

Any additional information below

I think the cause is the exception handling used in retry logic:

java-bigquery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryBaseService.java

Lines 29 to 35 in f4e5fc5

	public static final ExceptionHandler BIGQUERY_EXCEPTION_HANDLER =
	ExceptionHandler.newBuilder()
	.abortOn(RuntimeException.class)
	.retryOn(java.net.ConnectException.class) // retry on Connection Exception
	.retryOn(java.net.UnknownHostException.class) // retry on UnknownHostException
	.addInterceptors(EXCEPTION_HANDLER_INTERCEPTOR)
	.build();

Afaiu that code makes abort retries in front of a RuntimeException and BigQueryException is so.

EDIT: Just i have observed the interceptor above could make BigqueryException retryable but 403 code is not in the list of retryable errors:

java-bigquery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryException.java

Lines 39 to 41 in f4e5fc5

	private static final Set<Error> RETRYABLE_ERRORS =
	ImmutableSet.of(
	new Error(500, null), new Error(502, null), new Error(503, null), new Error(504, null));

I can see two alternatives if you think this should be fixed and the diagnostic is correct:

• open BigqueryOptions to allow users set a custom Exception handling
• Adds this case to the closed exception handler linked above

#1498 asked to retry on a similiar error and was fixed but the return code was 400 in that case (instead 403)

Many thanks in advance

[farhan0102]

From my investigation, Adding 403 to retry-able errors in BigQueryException is likely not going to resolve the problem. The 403 is unauthorized due to rate limiting. I believe the correct layer to handle this is at the application by adding delay or exponential back off with jitter. A retry at this level is too general for a 403 as there could be many reasons for it and in this case likely won't solve this problem due to a rolling window it can still hit.

The current code pattern is retry on Server Error Responses, we don't have any that does for Client Error Responses that do retry which makes me lean towards doing this at application layer vs at the exception handler.

Requested add: [403: Forbidden]

Current:
[500: Internal Server Error,
502: Bad Gateway,
503: Service Unavailable,
504: Gateway Timeout]

Please feel free to reopen if you disagree or have more to say about this issue.

[jneira-stratio]

@takayahilton thanks for the analysis and the info only want to note that:

• one of the alternatives i suggested was open BigqueryOptions to allow users set a custom Exception handling, what do you think about that path?
• there is existing logic to make retries rely on exception messages and not only http error codes, see for example:

java-bigquery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryImpl.java

Lines 242 to 247 in 24aac14

	private static final BigQueryRetryConfig DEFAULT_RETRY_CONFIG =
	BigQueryRetryConfig.newBuilder()
	.retryOnMessage(BigQueryErrorMessages.RATE_LIMIT_EXCEEDED_MSG)
	.retryOnMessage(BigQueryErrorMessages.JOB_RATE_LIMIT_EXCEEDED_MSG)
	.retryOnRegEx(BigQueryErrorMessages.RetryRegExPatterns.RATE_LIMIT_EXCEEDED_REGEX)
	.build(); // retry config with Error Messages and RegEx for RateLimitExceeded Error

In fact the pr i linked above added support for retries on 400 http code using a regex based approach to narrow appropiate errors related with quotas.
I would say that if the key part is make retries trigger on the excedded quota error, the http error code chosen by the server has less relevance

Thanks for your time, i am gonna reopen if you dont mind

EDIT: sorry but i dont see the button to reopen the issue, it seems issue author does not have permission to do it