Vision: '503 ServiceUnavailable Connect Failed' raised during 'annotate_image'

[DowneyTung]

1. OS type and version
   HostOS: MacOS High Sierra 10.13.6
   Docker Version: 18.06.10-ce
   Python version: 3.6

2. google-cloud-python version pip show google-cloud, pip show google-<service> or pip freeze
   Google-cloud version: 0.34.0

3. Stacktrace if available

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
return callable_(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 532, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 466, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Connect Failed"
debug_error_string = "{"created":"@1538572249.935821774","description":"Failed to create subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":2636,"referenced_errors":[{"created":"@1538572249.935744425","description":"Pick Cancelled","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":241,"referenced_errors":[{"created":"@1538572249.935691680","description":"Connect Failed","file":"src/core/ext/filters/client_channel/subchannel.cc","file_line":663,"grpc_status":14,"referenced_errors":[{"created":"@1538572249.935687130","description":"OS Error","errno":97,"file":"src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":283,"os_error":"Address family not supported by protocol","syscall":"socket","target_address":"[2607:f8b0:4006:803::200a]:443"}]}]}]}"


The above exception was the direct cause of the following exception:

File "/usr/local/lib/python3.6/site-packages/google/cloud/vision_helpers/decorators.py", line 111, in inner
response = self.annotate_image(request, retry=retry, timeout=timeout)
File "/usr/local/lib/python3.6/site-packages/google/cloud/vision_helpers/__init__.py", line 67, in annotate_image
r = self.batch_annotate_images([request], retry=retry, timeout=timeout)
File "/usr/local/lib/python3.6/site-packages/google/cloud/vision_v1/gapic/image_annotator_client.py", line 220, in batch_annotate_images
request, retry=retry, timeout=timeout, metadata=metadata)
File "/usr/local/lib/python3.6/site-packages/google/api_core/gapic_v1/method.py", line 139, in __call__
return wrapped_func(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 61, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.ServiceUnavailable: 503 Connect Failed

4. Steps to reproduce
   To reproduce it, we can try to instantiate multiple instances of google vision client and call the text detection service multiple times and in parallel. All these should be run inside docker containers
   Here is the small python snippet that you can use to run multiple times in parallel by instantiate multiple client instances and do text detection

def extract_text_from_image_using_google_vision_api(image_path):
    """Extract the text from a target image using google vision api
    Parameters:
    image_path: full image path ending with file name
    """
    client = vision.ImageAnnotatorClient()
    with io.open(image_path, 'rb') as image_file:
        content = image_file.read()
    image = types.Image(content=content)
    response = client.text_detection(image=image)
    texts = response.text_annotations
    return texts

[tseaver]

@DowneyTung Can you please show the version of google-cloud-vision you are using? I.e., the output from pip show google-cloud-vision.

[tseaver]

Odd... Unlike #6121, the Vision GAPIC configuration shows UNAVAILABLE as one of the retryable exception types for idempotent methods, and BatchAnnotateImages is marked idempotent.

I'm guessing that the API call must have been retried, but that the 503 error persisted beyond the count / timeout.

[DowneyTung]

@DowneyTung Can you please show the version of google-cloud-vision you are using? I.e., the output from pip show google-cloud-vision.
Hey @tseaver
The version is 0.34.0

[DowneyTung]

Odd... Unlike #6121, the Vision GAPIC configuration shows UNAVAILABLE as one of the retryable exception types for idempotent methods, and BatchAnnotateImages is marked idempotent.

I'm guessing that the API call must have been retried, but that the 503 error persisted beyond the count / timeout.

I am not sure I am following your comment. Do you need extra info for investigate this issue?
Thanks for your help

[tseaver]

Might be related to the service outage mentioned in #6172.

/cc @beccasaurus, @zinjiggle.

[DowneyTung]

Hi @tseaver Here is some more stacktrace for resolving this issue:

2018-10-19 13:11:41,079 google.auth.transport.requests DEBUG Making request: POST https://oauth2.googleapis.com/token
2018-10-19 13:11:41,082 urllib3.connectionpool DEBUG Starting new HTTPS connection (1): oauth2.googleapis.com:443
2018-10-19 13:11:41,222 grpc._plugin_wrapping ERROR AuthMetadataPluginCallback "<google.auth.transport.grpc.AuthMetadataPlugin object at 0x7f382aedb160>" raised exception!
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 56, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name does not resolve

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 849, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 314, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 180, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f382ad3f128>: Failed to establish a new connection: [Errno -2] Name does not resolve

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='oauth2.googleapis.com', port=443): Max retries exceeded with url: /token (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f382ad3f128>: Failed to establish a new connection: [Errno -2] Name does not resolve',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/google/auth/transport/requests.py", line 120, in __call__
    **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 513, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='oauth2.googleapis.com', port=443): Max retries exceeded with url: /token (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f382ad3f128>: Failed to establish a new connection: [Errno -2] Name does not resolve',))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/grpc/_plugin_wrapping.py", line 79, in __call__
    callback_state, callback))
  File "/usr/local/lib/python3.6/site-packages/google/auth/transport/grpc.py", line 77, in __call__
    callback(self._get_authorization_headers(context), None)
  File "/usr/local/lib/python3.6/site-packages/google/auth/transport/grpc.py", line 65, in _get_authorization_headers
    headers)
  File "/usr/local/lib/python3.6/site-packages/google/auth/credentials.py", line 122, in before_request
    self.refresh(request)
  File "/usr/local/lib/python3.6/site-packages/google/oauth2/service_account.py", line 322, in refresh
    request, self._token_uri, assertion)
  File "/usr/local/lib/python3.6/site-packages/google/oauth2/_client.py", line 145, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/usr/local/lib/python3.6/site-packages/google/oauth2/_client.py", line 106, in _token_endpoint_request
    method='POST', url=token_uri, headers=headers, body=body)
  File "/usr/local/lib/python3.6/site-packages/google/auth/transport/requests.py", line 124, in __call__
    six.raise_from(new_exc, caught_exc)
  File "<string>", line 3, in raise_from
google.auth.exceptions.TransportError: HTTPSConnectionPool(host='oauth2.googleapis.com', port=443): Max retries exceeded with url: /token (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f382ad3f128>: Failed to establish a new connection: [Errno -2] Name does not resolve',))
2018-10-19 13:11:41,719 google.auth.transport.requests DEBUG Making request: POST https://oauth2.googleapis.com/token

[DowneyTung]

@tseaver I think there is some connection issue, but I am not sure whether this is something related to google. I solved this issue by just retrying to instantiate a new client, usually it works after I instantiate a new client.
Here is my fix, which may help others who face similar issue:

from google.cloud import vision
from google.cloud.vision import types
import io

def extract_text_from_image_using_google_vision_api(image_path):
    with io.open(image_path, 'rb') as image_file:
        content = image_file.read()
        image = types.Image(content=content)

    # Given there are random errors when using google vision client, we want to try to use google vision api three times
    for i in range(3):
        try:
            client = vision.ImageAnnotatorClient()
            response = client.text_detection(image=image)
            texts = response.text_annotations
            return texts
        except Exception as ex:
            print(ex)
            print("Retrying {} time with google vision api...".format(i))

[tseaver]

@DowneyTung Interesting. The core error in that set of stacktraces is socket.gaierror: [Errno -2] Name does not resolve, indicates that DNS for the endpoint host (oauth2.googleapis.com) was failing. DNS failure could have a number of causes, none of which we can address in the google-cloud-vision library.

[DowneyTung]

failure could have a number of c

I would think it is not related to google-cloud-vision. Should I open a new ticket somewhere else for investigating this issue?

[tseaver]

@DowneyTung

Should I open a new ticket somewhere else for investigating this issue?

I'm not sure where that would be: maybe with your hosting provider (if their DNS infrastructure was flapping?) If you think the issue was with the "root" DNS entry for vision.googleapis.com, then perhaps one of the Vision API support resources?

[tanglion]

please check your local network.
I had the same problem. finally i found the reason is i run the script in local ,but the dns was not reached~!

[finoptimal-dev]

I see this happening too and believe it's ISP-specific.