TypeError when trying to issue BatchHttpRequest when service was built using application default credentials

[mgilson]

This issue is very similar to #211. My team has an app-engine application that uses batch requests in the Gmail API. After an upgrade of this client library, we started seeing failures:

...
  File "/base/data/home/apps/s~app-id/modname:version/path/to/my/code.py", line 241, in _user_method
    return batch_request.execute(http=self._http)
  File "/base/data/home/apps/s~app-id/modname:version/third_party/oauth2client/_helpers.py", line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/base/data/home/apps/s~app-id/modname:version/third_party/googleapiclient/http.py", line 1417, in execute
    self._execute(http, self._order, self._requests)
  File "/base/data/home/apps/s~app-id/modname:version/third_party/googleapiclient/http.py", line 1333, in _execute
    body = self._serialize_request(request)
  File "/base/data/home/apps/s~app-id/modname:version/third_party/googleapiclient/http.py", line 1204, in _serialize_request
    request.http.request.credentials.apply(headers)
  File "/base/data/home/apps/s~app-id/modname:version/third_party/oauth2client/client.py", line 558, in apply
    headers['Authorization'] = 'Bearer ' + self.access_token
TypeError: cannot concatenate 'str' and 'NoneType' objects

Our usage of this client library looks like the following:

from googleapiclient.discovery import build
gmail_service = build('gmail', 'v1')  # Uses application credentials

...

authorized_http = user_credentials.authorize(httplib2.Http())  # Uses end user credentials

batch_request = gmail_service.new_batch_http_request()
batch_request.add(
    gmail_service.users().threads().get(id=thread_id, userId='me'),
    callback=callback,
    request_id=thread_id)

batch_request.execute(http=authorized_http)

The gmail_service is cached at the application level to avoid needing to do an API call for each request (which is why it ends up with the application credentials). What I believe is happening is that

gmail_service.users().threads().get(id=thread_id, userId='me')

ends up creating an HttpRequest that uses the same http that was used to construct gmail_service (i.e. the application credentials) and I believe that the application credentials have no access_token. This results in the TypeError seen above. The fix in #232 won't help (the http that is getting passed to batch_request.execute is valid).

It seems to me that the application of credentials in _serialize_request should be conditional on the credentials actually having an access token:

if request.http is not None and hasattr(request.http.request,
    'credentials'):
    if request.http.request.credentials.access_token:
        request.http.request.credentials.apply(headers)

Otherwise, if the sub-request isn't authenticated then the authentication from the outer request should be used (at least, if the gmail docs are any indicator):

The HTTP headers for the outer batch request, except for the Content- headers such as Content-Type, apply to every request in the batch. If you specify a given HTTP header in both the outer request and an individual call, then the individual call header's value overrides the outer batch request header's value. The headers for an individual call apply only to that call.

For example, if you provide an Authorization header for a specific call, then that header applies only to that call. If you provide an Authorization header for the outer request, then that header applies to all of the individual calls unless they override it with Authorization headers of their own.

---

In the event that this behavior is actually working as intended or if my proposed fix isn't satisfactory and a real fix would be too difficult, a work-around to this issue is possible by updating the requests that you add to the batch with an appropriately authorized http instance. i.e changing the above to:

batch_request = gmail_service.new_batch_http_request()

request =  gmail_service.users().threads().get(id=thread_id, userId='me')
request.http = authorized_http  # explicitly set the authorization on the request.

batch_request.add(
    request,
    callback=callback,
    request_id=thread_id)

batch_request.execute(http=authorized_http)

Seems to fix the issue.

[theacodes]

@mgilson thanks for this really detailed bug report. Are you willing to send a PR to fix this issue? If so, I'll be happy to review.

[mgilson]

Sure, I can submit a PR. I've got a patch ready here, but I don't know anything about your test setup. I tried downloading tox in my virtual environment (pip install tox) and running it, but I seem to be running into all sorts of errors when I run it. The log file looks like this:

$ cat /Users/mgilson/git/google-api-python-client/.tox/py27-oauth2client1/log/py27-oauth2client1-0.log
actionid: py27-oauth2client1
msg: getenv
cmdargs: ['/Users/mgilson/anaconda/bin/python', '-m', 'virtualenv', '--python', '/Users/mgilson/anaconda/bin/python2.7', 'py27-oauth2client1']
env: {'TERM_PROGRAM_VERSION': '361.1', 'LOGNAME': 'mgilson', 'USER': 'mgilson', 'HOME': '/Users/mgilson', 'PATH': '/Users/mgilson/git/google-api-python-client/.tox/py27-oauth2client1/bin:/Users/mgilson/bin/google-cloud-sdk/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Users/mgilson/anaconda/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.5/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Users/mgilson/bin/scala-2.11.8/bin:/Users/mgilson/bin/spark-1.6.1-bin-hadoop2.6/bin:/Users/mgilson/bin', 'VIRTUALENVWRAPPER_SCRIPT': '/usr/local/bin/virtualenvwrapper.sh', 'DISPLAY': '/private/tmp/com.apple.launchd.Bxzp5coMLF/org.macosforge.xquartz:0', ... 'Apple_PubSub_Socket_Render': '/private/tmp/com.apple.launchd.rSbwT8LUMA/Render', 'PS1': '\\u:.../\\W \\[\x1b[36m\\]$git_branch\\[\x1b[31m\\]$git_dirty\\[\x1b(B\x1b[0m\\]$ ', '_': '/Users/mgilson/anaconda/bin/tox', 'VIRTUALENVWRAPPER_PROJECT_FILENAME': '.project', 'LSCOLORS': 'ExFxCxDxBxegedabagacad', 'VIRTUALENVWRAPPER_HOOK_DIR': '/Users/mgilson/Envs', 'OLDPWD': '/Users/mgilson/git', 'BASH_RC_SOURCED': '1', 'CLICOLOR': '1', '__CF_USER_TEXT_ENCODING': '0x1F5:0x0:0x0', 'PWD': '/Users/mgilson/git/google-api-python-client', 'VIRTUALENVWRAPPER_WORKON_CD': '1'}

New python executable in /Users/mgilson/git/google-api-python-client/.tox/py27-oauth2client1/bin/python2.7
Also creating executable in /Users/mgilson/git/google-api-python-client/.tox/py27-oauth2client1/bin/python
Installing setuptools, pip, wheel...
  Complete output from command /Users/mgilson/git/g...lient1/bin/python2.7 - setuptools pip wheel:
  Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
  File "/Users/mgilson/anaconda/lib/python2.7/tempfile.py", line 32, in <module>
    import io as _io
  File "/Users/mgilson/anaconda/lib/python2.7/io.py", line 51, in <module>
    import _io
ImportError: dlopen(/Users/mgilson/git/google-api-python-client/.tox/py27-oauth2client1/lib/python2.7/lib-dynload/_io.so, 2): Symbol not found: __PyCodecInfo_GetIncrementalDecoder
  Referenced from: /Users/mgilson/git/google-api-python-client/.tox/py27-oauth2client1/lib/python2.7/lib-dynload/_io.so
  Expected in: flat namespace
 in /Users/mgilson/git/google-api-python-client/.tox/py27-oauth2client1/lib/python2.7/lib-dynload/_io.so
----------------------------------------
...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
  File "/Users/mgilson/anaconda/lib/python2.7/site-packages/virtualenv.py", line 2328, in <module>
    main()
  File "/Users/mgilson/anaconda/lib/python2.7/site-packages/virtualenv.py", line 713, in main
    symlink=options.symlink)
  File "/Users/mgilson/anaconda/lib/python2.7/site-packages/virtualenv.py", line 945, in create_environment
    download=download,
  File "/Users/mgilson/anaconda/lib/python2.7/site-packages/virtualenv.py", line 901, in install_wheel
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
  File "/Users/mgilson/anaconda/lib/python2.7/site-packages/virtualenv.py", line 797, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /Users/mgilson/git/g...lient1/bin/python2.7 - setuptools pip wheel failed with error code 1
Running virtualenv with interpreter /Users/mgilson/anaconda/bin/python2.7

Have you ever seen anything like this before?

[snarfed]

this bit me recently when i upgraded to oauth2client 4.1.1 and google-api-python-client 1.6.2. patching #351 into my local lib fixes it. i'd love to see that PR merged so it gets into the next release!

[theacodes]

Resolved by #376.