Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DistributionNotFound error potentially caused by pkg_resources #1307

Closed
dionearle opened this issue Aug 9, 2022 · 2 comments
Closed

DistributionNotFound error potentially caused by pkg_resources #1307

dionearle opened this issue Aug 9, 2022 · 2 comments
Labels
api: bigquery Issues related to the googleapis/python-bigquery API. status: will not fix Invalid (untrue/unsound/erroneous), inconsistent with product, not on roadmap.

Comments

@dionearle
Copy link

Hi,

when using this package on my local machine I have no issues, however when deploying to our serverless environment in an AWS lambda we get the error seen at the bottom of this issue.

We have tried numerous changes, such as using the same version of pip being used locally, and disabling slim in the serverless python requirements, but the issue is still happening. Locally we use poetry to install packages.

When looking at the error message, the issue seems to be around pkg_resources, and from looking at other google packages I can see similar issues have been encountered and resolved around this. Is it possible to do something similar here? We are currently blocked from using this package until this issue is resolved.

Related issues from other projects:
googleapis/python-api-core#27
https://github.com/googleapis/google-api-python-client/pull/1623/files

Environment details

  • OS type and version: Linux on a docker container
  • Python version: 3.7.10
  • pip version: 22.0.4
  • google-cloud-bigquery version: 3.3.0

Steps to reproduce

  1. Deploy the build and attempt to run any code

Stack trace

{
  "errorMessage": "The 'google-cloud-bigquery-storage' distribution was not found and is required by the application",
  "errorType": "DistributionNotFound",
  "stackTrace": [
    "  File \"/var/lang/lib/python3.7/imp.py\", line 234, in load_module\n    return load_source(name, filename, file)\n",
    "  File \"/var/lang/lib/python3.7/imp.py\", line 171, in load_source\n    module = _load(spec)\n",
    "  File \"<frozen importlib._bootstrap>\", line 696, in _load\n",
    "  File \"<frozen importlib._bootstrap>\", line 677, in _load_unlocked\n",
    "  File \"<frozen importlib._bootstrap_external>\", line 728, in exec_module\n",
    "  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n",
    "  File \"/opt/python/lib/python3.7/site-packages/datadog_lambda/handler.py\", line 27, in <module>\n    handler_module = import_module(mod_name)\n",
    "  File \"/var/lang/lib/python3.7/importlib/__init__.py\", line 127, in import_module\n    return _bootstrap._gcd_import(name[level:], package, level)\n",
    "  File \"<frozen importlib._bootstrap>\", line 1006, in _gcd_import\n",
    "  File \"<frozen importlib._bootstrap>\", line 983, in _find_and_load\n",
    "  File \"/opt/python/lib/python3.7/site-packages/ddtrace/internal/import_hooks.py\", line 215, in wrapped_find_and_load_unlocked\n    return exec_and_call_hooks(module_name, wrapped, args, kwargs)\n",
    "  File \"/opt/python/lib/python3.7/site-packages/ddtrace/internal/import_hooks.py\", line 171, in exec_and_call_hooks\n    return wrapped(*args, **kwargs)\n",
    "  File \"<frozen importlib._bootstrap>\", line 967, in _find_and_load_unlocked\n",
    "  File \"<frozen importlib._bootstrap>\", line 677, in _load_unlocked\n",
    "  File \"<frozen importlib._bootstrap_external>\", line 728, in exec_module\n",
    "  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n",
    "  File \"/var/task/handlers/functions/voz_rf.py\", line 10, in <module>\n    from google.cloud.bigquery import Client\n",
    "  File \"<frozen importlib._bootstrap>\", line 983, in _find_and_load\n",
    "  File \"/opt/python/lib/python3.7/site-packages/ddtrace/internal/import_hooks.py\", line 215, in wrapped_find_and_load_unlocked\n    return exec_and_call_hooks(module_name, wrapped, args, kwargs)\n",
    "  File \"/opt/python/lib/python3.7/site-packages/ddtrace/internal/import_hooks.py\", line 171, in exec_and_call_hooks\n    return wrapped(*args, **kwargs)\n",
    "  File \"/tmp/sls-py-req/google/cloud/bigquery/__init__.py\", line 35, in <module>\n    from google.cloud.bigquery.client import Client\n",
    "  File \"<frozen importlib._bootstrap>\", line 983, in _find_and_load\n",
    "  File \"/opt/python/lib/python3.7/site-packages/ddtrace/internal/import_hooks.py\", line 215, in wrapped_find_and_load_unlocked\n    return exec_and_call_hooks(module_name, wrapped, args, kwargs)\n",
    "  File \"/opt/python/lib/python3.7/site-packages/ddtrace/internal/import_hooks.py\", line 171, in exec_and_call_hooks\n    return wrapped(*args, **kwargs)\n",
    "  File \"/tmp/sls-py-req/google/cloud/bigquery/client.py\", line 59, in <module>\n    from google.cloud.bigquery_storage_v1.services.big_query_read.client import (\n",
    "  File \"<frozen importlib._bootstrap>\", line 983, in _find_and_load\n",
    "  File \"/opt/python/lib/python3.7/site-packages/ddtrace/internal/import_hooks.py\", line 215, in wrapped_find_and_load_unlocked\n    return exec_and_call_hooks(module_name, wrapped, args, kwargs)\n",
    "  File \"/opt/python/lib/python3.7/site-packages/ddtrace/internal/import_hooks.py\", line 171, in exec_and_call_hooks\n    return wrapped(*args, **kwargs)\n",
    "  File \"<frozen importlib._bootstrap>\", line 983, in _find_and_load\n",
    "  File \"/opt/python/lib/python3.7/site-packages/ddtrace/internal/import_hooks.py\", line 215, in wrapped_find_and_load_unlocked\n    return exec_and_call_hooks(module_name, wrapped, args, kwargs)\n",
    "  File \"/opt/python/lib/python3.7/site-packages/ddtrace/internal/import_hooks.py\", line 171, in exec_and_call_hooks\n    return wrapped(*args, **kwargs)\n",
    "  File \"<frozen importlib._bootstrap>\", line 983, in _find_and_load\n",
    "  File \"/opt/python/lib/python3.7/site-packages/ddtrace/internal/import_hooks.py\", line 215, in wrapped_find_and_load_unlocked\n    return exec_and_call_hooks(module_name, wrapped, args, kwargs)\n",
    "  File \"/opt/python/lib/python3.7/site-packages/ddtrace/internal/import_hooks.py\", line 171, in exec_and_call_hooks\n    return wrapped(*args, **kwargs)\n",
    "  File \"<frozen importlib._bootstrap>\", line 983, in _find_and_load\n",
    "  File \"/opt/python/lib/python3.7/site-packages/ddtrace/internal/import_hooks.py\", line 215, in wrapped_find_and_load_unlocked\n    return exec_and_call_hooks(module_name, wrapped, args, kwargs)\n",
    "  File \"/opt/python/lib/python3.7/site-packages/ddtrace/internal/import_hooks.py\", line 171, in exec_and_call_hooks\n    return wrapped(*args, **kwargs)\n",
    "  File \"/tmp/sls-py-req/google/cloud/bigquery_storage_v1/__init__.py\", line 22, in <module>\n    \"google-cloud-bigquery-storage\"\n",
    "  File \"/opt/python/lib/python3.7/site-packages/pkg_resources/__init__.py\", line 481, in get_distribution\n    dist = get_provider(dist)\n",
    "  File \"/opt/python/lib/python3.7/site-packages/pkg_resources/__init__.py\", line 357, in get_provider\n    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]\n",
    "  File \"/opt/python/lib/python3.7/site-packages/pkg_resources/__init__.py\", line 900, in require\n    needed = self.resolve(parse_requirements(requirements))\n",
    "  File \"/opt/python/lib/python3.7/site-packages/pkg_resources/__init__.py\", line 786, in resolve\n    raise DistributionNotFound(req, requirers)\n"
  ]
}
@dionearle
Copy link
Author

I managed to create a fix for this in my own fork as seen in this commit.

Would we be able to make a similar change in the main fork? If so I can create a pull request if the change is fine as is, otherwise it'd be great to have this implemented properly so we can go back to using the main version.

@chalmerlowe
Copy link
Collaborator

In looking at this library, back in Nov 2022, a change was made to the python-bigquery-storage library to change how versions are derived. If that change has not provided relief, please feel free to re-open this ticket. Otherwise, gonna close this issue.

Relevant change: https://github.com/googleapis/python-bigquery-storage/blob/49834632ce42545bc6cfd9f56fcd8b230a64627d/google/cloud/bigquery_storage_v1/__init__.py#L19-L21

@chalmerlowe chalmerlowe added the status: will not fix Invalid (untrue/unsound/erroneous), inconsistent with product, not on roadmap. label Feb 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery API. status: will not fix Invalid (untrue/unsound/erroneous), inconsistent with product, not on roadmap.
Projects
None yet
Development

No branches or pull requests

2 participants