Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exception Value: api_request() got an unexpected keyword argument 'extra_api_info' #831

Closed
RomainFayolle opened this issue Aug 4, 2022 · 14 comments
Assignees
Labels
api: storage Issues related to the googleapis/python-storage API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: question Request for information or clarification. Not an issue.

Comments

@RomainFayolle
Copy link

Hi,

I'm having an issue on google-cloud-storage 2.3.0, I also tried it with 2.5.0. I use Django and I have a file model that is linked to GCS. When getting the size of the file I have an error: Exception Value: api_request() got an unexpected keyword argument 'extra_api_info'
I fixed the issue by downgrading google-cloud-storage to 1.44.0 as suggested in this stack overflow issue

Environment details

  • OS: Ubuntu 20.04.2 LTS
  • Python version: 3.8.10
  • pip version: 19.0.3
  • google-cloud-storage version: 2.3.0 and 2.5.0
  • Django: 3.2.13
  • Django Rest Framework: 3.13.1

Steps to reproduce

  1. Create a Django model
  2. Set a size @Property
  3. Try to get the size property

Code example

class MyFileClass(models.Model):
    file = models.FileField(
        verbose_name=_('File'),
        upload_to='path_to_upload',
        null=False
    )

    @property
    def size(self):
        return self.file.size if self.file else None

Stack trace

Traceback (most recent call last):
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/mixins.py", line 43, in list
    return self.get_paginated_response(serializer.data)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/serializers.py", line 768, in data
    ret = super().data
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/serializers.py", line 253, in data
    self._data = self.to_representation(self.instance)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/serializers.py", line 686, in to_representation
    return [
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/serializers.py", line 687, in <listcomp>
    self.child.to_representation(item) for item in iterable
  File "/home/ubuntu/my_project/my_app/serializers.py", line 65, in to_representation
    return super().to_representation(instance)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/serializers.py", line 509, in to_representation
    attribute = field.get_attribute(instance)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/fields.py", line 457, in get_attribute
    return get_attribute(instance, self.source_attrs)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/fields.py", line 97, in get_attribute
    instance = getattr(instance, attr)
  File "/home/ubuntu/my_project/my_app/models.py", line 181, in size
    return self.file.size if self.file else None
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/django/db/models/fields/files.py", line 71, in size
    return self.storage.size(self.name)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/storages/backends/gcloud.py", line 246, in size
    blob = self._get_blob(name)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/storages/backends/gcloud.py", line 237, in _get_blob
    blob = self.bucket.get_blob(name)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/google/cloud/storage/bucket.py", line 1218, in get_blob
    blob.reload(
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/google/cloud/storage/_helpers.py", line 238, in reload
    api_response = client._get_resource(
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/google/cloud/storage/client.py", line 366, in _get_resource
    return self._connection.api_request(
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/google/cloud/storage/_http.py", line 73, in api_request
    return call()
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    return retry_target(
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/google/api_core/retry.py", line 189, in retry_target
    return target()

Exception Type: TypeError at /my_view
Exception Value: api_request() got an unexpected keyword argument 'extra_api_info'

Everything is done under the hood, I'm not using api_request() myself. Django is when trying to get the size parameter of my file by using google-cloud-storage.
CC @RignonNoel

@product-auto-label product-auto-label bot added the api: storage Issues related to the googleapis/python-storage API. label Aug 4, 2022
@unforced
Copy link
Contributor

unforced commented Aug 4, 2022

Oh yikes, this looks related to my change

#741
googleapis/python-cloud-core#183

Noting that this has also been reported here:
https://stackoverflow.com/questions/72306979/client-get-bucket-returns-error-api-request-got-an-unexpected-keyword-argum

Researching this further and will update here as soon as I discover more or have a fix in place.

@andrewsg andrewsg added the priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. label Aug 4, 2022
@unforced
Copy link
Contributor

unforced commented Aug 8, 2022

Just to be clear, does this happen every-time for you? Or is it intermittent?

@unforced
Copy link
Contributor

unforced commented Aug 8, 2022

Could you also run a pip freeze and share the output?

@RomainFayolle
Copy link
Author

Hi !
It happens every time. My use case is: I have a model made for files. I have a front end listing and displaying those files. When the front call Django Rest API to get the data to display, Django get all attributes for those files, and especially the size property, triggering the error every time. In the end the back always send the error you see in my post.

For the pip freeze:

asgiref==3.5.2
billiard==3.6.4.0
CacheControl==0.12.6
cachetools==4.2.2
celery==5.2.6
certifi==2021.10.8
cffi==1.14.5
chardet==4.0.0
click==8.1.3
click-didyoumean==0.0.3
click-plugins==1.1.1
click-repl==0.2.0
colorama==0.4.3
coreapi==2.3.3
coreschema==0.0.4
defusedxml==0.7.1
diff-match-patch==20200713
dj-database-url==0.5.0
Django==3.2.13
django-admin-autocomplete-filter==0.7.1
django-anymail==8.5
django-celery-beat==2.2.1
django-celery-results==2.3.1
django-cors-headers==3.12.0
django-filter==21.1
django-import-export==2.8.0
django-storages==1.12.3
django-timezone-field==4.2.3
djangorestframework==3.13.1
et-xmlfile==1.1.0
factory-boy==3.2.1
Faker==8.4.0
firebase-admin==5.0.2
future==0.18.2
ghp-import==2.1.0
google-api-core==1.31.5
google-api-python-client==2.17.0
google-auth==1.30.1
google-auth-httplib2==0.1.0
google-cloud-core==1.6.0
google-cloud-datastore==2.5.1
google-cloud-firestore==2.2.0
google-cloud-monitoring==2.9.2
google-cloud-storage==1.44.0
google-cloud-trace==0.24.0
google-crc32c==1.1.2
google-resumable-media==2.3.3
googleapis-common-protos==1.53.0
grpcio==1.38.0
grpcio-status==1.46.3
gunicorn==20.0.4
httplib2==0.19.1
idna==2.10
importlib-metadata==4.11.4
itypes==1.2.0
Jinja2==3.0.1
joblib==1.0.1
kombu==5.2.4
livereload==2.6.3
lunr==0.5.8
Markdown==3.3.4
MarkupPy==1.14
MarkupSafe==2.0.1
mergedeep==1.3.4
mkdocs==1.3.0
mkdocs-material==5.2.3
mkdocs-material-extensions==1.0.1
msgpack==1.0.2
nltk==3.6.2
odfpy==1.4.1
opencensus==0.9.0
opencensus-context==0.1.2
opencensus-ext-django==0.7.6
opencensus-ext-stackdriver==0.8.0
openpyxl==3.0.7
packaging==20.9
prompt-toolkit==3.0.19
proto-plus==1.19.0
protobuf==3.17.1
psycopg2-binary==2.8.6
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
Pygments==2.9.0
pymdown-extensions==8.2
pyparsing==2.4.7
python-crontab==2.6.0
python-dateutil==2.8.1
python-decouple==3.6
pytz==2022.1
PyYAML==5.4.1
pyyaml-env-tag==0.1
regex==2021.4.4
requests==2.25.1
requests-futures==1.0.0
rsa==4.7.2
six==1.16.0
sqlparse==0.4.2
tablib==3.0.0
text-unidecode==1.3
tornado==6.1
tqdm==4.40.2
uritemplate==3.0.1
urllib3==1.26.5
vine==5.0.0
watchdog==2.1.8
wcwidth==0.2.5
xlrd==2.0.1
XlsxWriter==1.3.3
xlwt==1.3.0
zipp==3.8.0```

@unforced
Copy link
Contributor

unforced commented Aug 9, 2022 via email

@RomainFayolle
Copy link
Author

Haha yes sorry I was discussing with my colleague and I realized I shared the working version with 1.44.0, you were faster than me. Here is the freeze with 2.5.0:

asgiref==3.5.2
billiard==3.6.4.0
CacheControl==0.12.6
cachetools==4.2.2
celery==5.2.6
certifi==2021.10.8
cffi==1.14.5
chardet==4.0.0
click==8.1.3
click-didyoumean==0.0.3
click-plugins==1.1.1
click-repl==0.2.0
colorama==0.4.3
coreapi==2.3.3
coreschema==0.0.4
defusedxml==0.7.1
diff-match-patch==20200713
dj-database-url==0.5.0
Django==3.2.13
django-admin-autocomplete-filter==0.7.1
django-anymail==8.5
django-celery-beat==2.2.1
django-celery-results==2.3.1
django-cors-headers==3.12.0
django-filter==21.1
django-import-export==2.8.0
django-storages==1.12.3
django-timezone-field==4.2.3
djangorestframework==3.13.1
et-xmlfile==1.1.0
factory-boy==3.2.1
Faker==8.4.0
firebase-admin==5.0.2
future==0.18.2
ghp-import==2.1.0
google-api-core==1.31.5
google-api-python-client==2.17.0
google-auth==1.30.1
google-auth-httplib2==0.1.0
google-cloud-core==2.3.2
google-cloud-datastore==2.5.1
google-cloud-firestore==2.2.0
google-cloud-monitoring==2.9.2
google-cloud-storage==2.5.0
google-cloud-trace==0.24.0
google-crc32c==1.1.2
google-resumable-media==2.3.3
googleapis-common-protos==1.53.0
grpcio==1.38.0
grpcio-status==1.46.3
gunicorn==20.0.4
httplib2==0.19.1
idna==2.10
importlib-metadata==4.11.4
itypes==1.2.0
Jinja2==3.0.1
joblib==1.0.1
kombu==5.2.4
livereload==2.6.3
lunr==0.5.8
Markdown==3.3.4
MarkupPy==1.14
MarkupSafe==2.0.1
mergedeep==1.3.4
mkdocs==1.3.0
mkdocs-material==5.2.3
mkdocs-material-extensions==1.0.1
msgpack==1.0.2
nltk==3.6.2
odfpy==1.4.1
opencensus==0.9.0
opencensus-context==0.1.2
opencensus-ext-django==0.7.6
opencensus-ext-stackdriver==0.8.0
openpyxl==3.0.7
packaging==20.9
prompt-toolkit==3.0.19
proto-plus==1.19.0
protobuf==3.17.1
psycopg2-binary==2.8.6
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
Pygments==2.9.0
pymdown-extensions==8.2
pyparsing==2.4.7
python-crontab==2.6.0
python-dateutil==2.8.1
python-decouple==3.6
pytz==2022.1
PyYAML==5.4.1
pyyaml-env-tag==0.1
regex==2021.4.4
requests==2.25.1
requests-futures==1.0.0
rsa==4.7.2
six==1.16.0
sqlparse==0.4.2
tablib==3.0.0
text-unidecode==1.3
tornado==6.1
tqdm==4.40.2
uritemplate==3.0.1
urllib3==1.26.5
vine==5.0.0
watchdog==2.1.8
wcwidth==0.2.5
xlrd==2.0.1
XlsxWriter==1.3.3
xlwt==1.3.0
zipp==3.8.0

@unforced
Copy link
Contributor

Hey @RomainFayolle, just want you to know we're still trying to figure this issue out. Have not yet been able to reproduce the issue and when we try to install the requirements as you have them here (putting these in requirements.txt and running pip install -r requirements.txt), we're met with a strange conflict

The conflict is caused by:
    The user requested google-api-core==1.31.5
    google-api-python-client 2.17.0 depends on google-api-core<3.0.0dev and >=1.21.0
    google-cloud-core 2.3.2 depends on google-api-core!=2.0.*, !=2.1.*, !=2.2.*, !=2.3.0, <3.0.0dev and >=1.31.6

Continuing to investigate.

@unforced
Copy link
Contributor

@RomainFayolle would it be possible to share your requirements.txt file?

@unforced
Copy link
Contributor

What would also be helpful is if you could try putting the output from your pip freeze into a fresh environment requirements.txt and see if you're able to reproduce it. We still haven't been able to reproduce this on our end.

@adampilz
Copy link

adampilz commented Aug 23, 2022

Also found this error. STEPS:

  1. start up a vertex workbench notebook with TF enterprise 2.6, no GPU
  2. pip install tensorflow-data-validation
  3. pip install apache-beam[interactive]
  4. then in code :
    from google.cloud import storage
    PROJECT_ID = ""
    GCS_BUCKET = ""
    gcs_client = storage.Client(project=PROJECT_ID)
    bucket = gcs_client.get_bucket(GCS_BUCKET)

produces the error.

@RignonNoel
Copy link

@RomainFayolle would it be possible to share your requirements.txt file?

FYI Romain is in a middle of a holliday trip, it should be back soon and I'm sure he will take the time to update the issue with your requested items.

@RomainFayolle
Copy link
Author

RomainFayolle commented Sep 8, 2022

Hi @Breathtender sorry for the very late response. I was in a big trip and then lots of stuff to work on.
We tackle the issue with @RignonNoel and here is what we found:
I created a new venv and installed the dependencies:

amqp==5.1.1
asgiref==3.5.2
billiard==3.6.4.0
CacheControl==0.12.11
cachetools==5.2.0
celery==5.2.6
certifi==2021.10.8
charset-normalizer==2.1.1
click==8.1.3
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.2.0
colorama==0.4.3
coreapi==2.3.3
coreschema==0.0.4
defusedxml==0.7.1
diff-match-patch==20200713
dj-database-url==0.5.0
Django==3.2.13
django-admin-autocomplete-filter==0.7.1
django-anymail==8.5
django-celery-beat==2.2.1
django-celery-results==2.3.1
django-cors-headers==3.12.0
django-filter==21.1
django-import-export==2.8.0
django-storages==1.12.3
django-timezone-field==4.2.3
djangorestframework==3.13.1
et-xmlfile==1.1.0
factory-boy==3.2.1
Faker==14.2.0
firebase-admin==5.0.2
ghp-import==2.1.0
google-api-core==1.33.0
google-api-python-client==2.60.0
google-auth==2.11.0
google-auth-httplib2==0.1.0
google-cloud-core==2.3.2
google-cloud-datastore==2.5.1
google-cloud-firestore==2.6.1
google-cloud-monitoring==2.11.1
google-cloud-storage==2.5.0
google-cloud-trace==0.24.2
google-crc32c==1.5.0
google-resumable-media==2.3.3
googleapis-common-protos==1.56.4
grpcio==1.48.1
grpcio-status==1.48.1
gunicorn==20.0.4
httplib2==0.20.4
idna==3.3
importlib-metadata==4.12.0
itypes==1.2.0
Jinja2==3.1.2
kombu==5.2.4
Markdown==3.4.1
MarkupPy==1.14
MarkupSafe==2.1.1
mergedeep==1.3.4
mkdocs==1.3.0
mkdocs-material==5.2.3
mkdocs-material-extensions==1.0.3
msgpack==1.0.4
odfpy==1.4.1
opencensus==0.9.0
opencensus-context==0.1.3
opencensus-ext-django==0.7.6
opencensus-ext-stackdriver==0.8.0
openpyxl==3.0.10
packaging==21.3
prompt-toolkit==3.0.31
proto-plus==1.22.1
protobuf==3.20.1
psycopg2-binary==2.8.6
pyasn1==0.4.8
pyasn1-modules==0.2.8
Pygments==2.13.0
pymdown-extensions==9.5
pyparsing==3.0.9
python-crontab==2.6.0
python-dateutil==2.8.2
python-decouple==3.6
pytz==2022.1
PyYAML==6.0
pyyaml-env-tag==0.1
requests==2.28.1
requests-futures==1.0.0
rsa==4.9
six==1.16.0
sqlparse==0.4.2
tablib==3.2.1
tqdm==4.40.2
uritemplate==4.1.1
urllib3==1.26.12
vine==5.0.0
watchdog==2.1.9
wcwidth==0.2.5
xlrd==2.0.1
XlsxWriter==1.3.3
xlwt==1.3.0
zipp==3.8.1

Now the code is working as expected.
The issue was that we were able to update the google-cloud-storage without triggering any conflict and without it updating google-api-core even though if we install them separately we can see dependencies conflicts (As you show in your previous comment).
I think there is a missing warning or handling of dependencies on update ?

@harry-uglow
Copy link

harry-uglow commented Feb 8, 2023

@WildSunLove has there been any progress on fixing the underlying issue that causes this? We're encountering the same issue and are stuck using google-cloud-storage~=1.44.0. When we upgrade to 2.5.0 the following code encounters the same issue:

from google.cloud import storage

storage_client = storage.Client()
storage_client.list_blobs(bucket_name)

UPDATE: It's possible to use newer google-cloud-storage versions but there are incompatibilities that pip doesn't appear to warn you about. We fixed this by adding google-cloud-core~=2.3.0 to our requirements.txt - hope this helps.

@meredithslota meredithslota assigned andrewsg and unassigned unforced Mar 10, 2023
@meredithslota meredithslota added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. priority: p2 Moderately-important priority. Fix may not be included in next release. and removed priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. labels Mar 10, 2023
@danielduhh danielduhh assigned cojenco and unassigned andrewsg Apr 3, 2023
@cojenco
Copy link
Contributor

cojenco commented Apr 3, 2023

For more context, the argument extra_api_info was added in google-cloud-storage v2.3.0 and google-cloud-core v2.3.0 to allow extra information to be tracked in the X-Goog-API-Client header. The setup.py files (linked above) include updated dependencies versions and handle conflict warnings.

We have not been able to reproduce the issue as we see conflict warnings when running the following in a fresh venv respectively.

google-cloud-storage==2.5.0
google-cloud-core==1.6.0

The conflict is caused by:
    The user requested google-cloud-core==1.6.0
    google-cloud-storage 2.5.0 depends on google-cloud-core<3.0dev and >=2.3.0
google-cloud-storage==2.5.0
google-api-core==1.31.5
google-cloud-core==2.3.2

The conflict is caused by:
    The user requested google-api-core==1.31.5
    google-cloud-storage 2.5.0 depends on google-api-core!=2.0.*, !=2.1.*, !=2.2.*, !=2.3.0, <3.0.0dev and >=1.31.5
    google-cloud-core 2.3.2 depends on google-api-core!=2.0.*, !=2.1.*, !=2.2.*, !=2.3.0, <3.0.0dev and >=1.31.6

A recommendation is to explicitly specify/upgrade the following 3 libraries with current versions

google-cloud-storage==2.3.0
google-cloud-core==2.3.0
google-api-core==2.11.0

or the released (and later) versions of stated changes in #741 and googleapis/python-cloud-core#183

google-cloud-storage==2.3.0
google-cloud-core==2.3.0
google-api-core==1.34.0

If you still encounter this issue, please reopen and comment with the exact version of libraries you are using (for instance with pip freeze) and your OS/platform.

@cojenco cojenco added type: question Request for information or clarification. Not an issue. and removed type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Apr 3, 2023
@cojenco cojenco closed this as completed Apr 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the googleapis/python-storage API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: question Request for information or clarification. Not an issue.
Projects
None yet
Development

No branches or pull requests

8 participants