Datastore: 'ServiceUnavailable: 503 Connect Failed' raised during commit

[maxbellec]

During our CI (run by Strider CD in a container), we put some entities on the Datastore. For reasons I don't understand, it often fails (~30% of the time) with a 503 error. Re-testing (relaunching a new container and running the tests) fixes it.
I'm not sure #5410 would fix this issue, since retrying with try/except in the code doesn't solve this.

versions

google-api-core==1.1.1
google-api-python-client==1.6.5
google-auth==1.4.1
google-cloud-core==0.28.1
google-cloud-storage==1.8.0
google-resumable-media==0.3.1
googleapis-common-protos==1.5.3

stacktrace

  File "/home/feeder/lib/database.py", line 213, in db_upsert_multi
    datastore_client.put_multi(entities)  # Commit
  File "/usr/local/lib/python3.6/site-packages/google/cloud/datastore/client.py", line 431, in put_multi
    current.commit()
  File "/usr/local/lib/python3.6/site-packages/google/cloud/datastore/batch.py", line 273, in commit
    self._commit()
  File "/usr/local/lib/python3.6/site-packages/google/cloud/datastore/batch.py", line 249, in _commit
    self.project, mode, self._mutations, transaction=self._id)
  File "/usr/local/lib/python3.6/site-packages/google/cloud/datastore_v1/gapic/datastore_client.py", line 426, in commit
    request, retry=retry, timeout=timeout, metadata=metadata)
  File "/usr/local/lib/python3.6/site-packages/google/api_core/gapic_v1/method.py", line 139, in __call__
    return wrapped_func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/google/api_core/retry.py", line 260, in retry_wrapped_func
    on_error=on_error,
  File "/usr/local/lib/python3.6/site-packages/google/api_core/retry.py", line 177, in retry_target
    return target()
  File "/usr/local/lib/python3.6/site-packages/google/api_core/timeout.py", line 206, in func_with_timeout
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 61, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.ServiceUnavailable: 503 Connect Failed

[DowneyTung]

I have the same issue with google vision api.

[tseaver]

@maxbellec The traceback shows that the Commit API call was being funneled through the GAPIC retry machinery, but the exception type is not listed as being retryable in datastore_client_config.py: because the Commit method is marked non_idempotent, no exception types can be retried.

@DowneyTung please open a separate issue for the Vision API, showing your own traceback. Apologies: I see that you already opened #6190.

[DowneyTung]

@tseaver cool..thanks.. :)

[maxbellec]

@tseaver thank you for looking into my issue. I'm still unsure about why that errors happens quite often in the first place. I'm especially surprised that it happens only in the CI tests, do you have any guess on what I could try?

[tseaver]

@pcostell Can you suggest why the 503 is being raised from the Datastore API here? Should we be retrying the Commit call in that case?

[tseaver]

Needs help from @pcostell or another resource from the back-end Datastore team.

[corradio]

@maxbellec I think you can close this issue.
I'm working on the same team as @maxbellec and it turns out the error we got was because we were running against an emulator that wasn't fully started.
@tseaver you can also close the issue if needed.

[tseaver]

@corradio Thanks very much for the follow up.