[BigQuery] Retry on rateLimitExceeded response

[championj-foxtel]

Describe the feature

On a response of 403 rateLimitExceeded from BigQuery the tasks should retry if there are tries remaining

Additional context

rateLimitExceeded error in BigQuery returns a 403 instead of 429 https://cloud.google.com/bigquery/docs/error-messages

See similar discussion for google-cloud-python for some context: googleapis/google-cloud-python#6434

Who will this benefit?

When a temporary rate limit has been exceeded a retry here will prevent the pipeline from failling

Are you interested in contributing this feature?

I have patched this locally just checking for Forbidden response type; this should be improved to inspect error reason for 'rateLimitExceeded' or error message for 'Exceeded rate limits' or similar filter to only retry in rate limit exceeded scenario.

# plugins/bigquery/dbt/adapters/bigquery/connections.py
def _is_retryable(error):
    """Return true for errors that are unlikely to occur again if retried."""
    if isinstance(error, RETRYABLE_ERRORS):
        return True
    elif isinstance(error, google.api_core.exceptions.Forbidden) and any(e['reason'] == 'rateLimitExceeded' for e in error.errors):
        return True
    return False

[drewbanin]

wow - thanks for the heads up @championj-foxtel! I did not know that BQ returns a 403 here.... that is quite a decision.

Your local patch looks really great! If you'd be interested in opening a PR for that change, I think we'd be really happy to merge it. Let us know if that's something you'd be interested in - happy to help out if so!

[championj-foxtel]

Hi @drewbanin - that sounds great I will raise a PR for this one!

[jtcohen6]

Resolved by #2796