Don't use pkg_resources.get_distribution(..).version

[PureTryOut]

Environment details

• OS type and version: Alpine Linux edge
• Python version: 3.8
• pip version: Irrelevant, pure system packages used
• google-api-core version: 1.16.0

Issue

I'm trying to run the tests with the end goal being packaging this for Alpine Linux.
All the dependencies are installed and I'm running the tests with PYTHONPATH="$PWD/build/lib" pytest so I can run the tests without having the package installed yet.

I'm probably just doing something wrong, but every test file fails with the following:

_____________________________________________ ERROR collecting tests/unit/test_bidi.py ______________________________________________
tests/unit/test_bidi.py:24: in <module>
    from google.api_core import bidi
build/lib/google/api_core/__init__.py:23: in <module>
    __version__ = get_distribution("google-api-core").version
/usr/lib/python3.8/site-packages/pkg_resources/__init__.py:482: in get_distribution
    dist = get_provider(dist)
/usr/lib/python3.8/site-packages/pkg_resources/__init__.py:358: in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
/usr/lib/python3.8/site-packages/pkg_resources/__init__.py:901: in require
    needed = self.resolve(parse_requirements(requirements))
/usr/lib/python3.8/site-packages/pkg_resources/__init__.py:787: in resolve
    raise DistributionNotFound(req, requirers)
E   pkg_resources.DistributionNotFound: The 'google-api-core' distribution was not found and is required by the application

[busunkim96]

Hi - could you provide details on how you're installing google-api-core?

[PureTryOut]

I'm making an Alpine package with the intention to upstream it to their repositories.

The build script should speak for itself:

# Contributor: Bart Ribbers <[email protected]>
# Maintainer: Bart Ribbers <[email protected]>
pkgname=py3-google-api-core
pkgver=1.16.0
pkgrel=0
pkgdesc="Google Cloud Client Library for Python"
url="https://github.com/googleapis/google-cloud-python"
arch="noarch"
license="Apache-2.0"
depends="python3 py3-tz py3-requests py3-google-auth py3-protobuf py3-googleapis-common-protos"
makedepends="py3-setuptools py3-pytest py3-mock py3-grpcio"
source="https://pypi.python.org/packages/source/g/google-api-core/google-api-core-$pkgver.tar.gz"
builddir="$srcdir/google-api-core-$pkgver"

build() {
	python3 setup.py build
}

check() {
	PYTHONPATH="$PWD/build/lib" pytest
}

package() {
	python3 setup.py install --prefix=/usr --root="$pkgdir"
}

sha512sums="4897abf3ae59f84b6f0fa83c4704696649058028ed3d7439b2372cc9659634bd0f8eee2bc7f0a5fa7dce06f7e47581eeded659b6cb545a4e40d05de0e060d8f7  google-api-core-1.16.0.tar.gz"

I'm first building the package using setup.py build, after which I run pytest on the result.

[busunkim96]

Ah I don't think pkg_resources.get_distribution works correctly when the library is built with python3 setup.py ....

In our tests we always install the library with pip, even for local installs.

pip install .

python-api-core/noxfile.py

Line 33 in 335c109

session.install("-e", ".")

Is there some way to use pip?

[PureTryOut]

Not for installation sadly no, our packaging standards don't allow it.

[busunkim96]

Unfortunately we have to be a bit particular about maintaining the version in one place because it needs to be reported in request headers. We currently maintain a single source of truth by placing the version in setup.py.

We could explore moving to another way of single-sourcing the version if more demand. https://packaging.python.org/guides/single-sourcing-package-version/

I'll keep this issue open as a feature request.

[PureTryOut]

Unfortunately we have to be a bit particular about maintaining the version in one place because it needs to be reported in request headers.

That I understand. I'm sure more packages have this problem, I wonder how others do it.

[pganssle]

FWIW, both pkg_resources and directly invoking setup.py install are semi-deprecated and are mostly unsupported.

The replacement for pkg_resources.distribution(...).version is importlib.metadata.version(...) (backported to pre-3.7 Python as importlib_metadata), though I doubt that will solve anyone's problem if the installation metadata is simply not present.

More context about the setup.py install deprecation here, but the short answer is that you should always prefer pip install to setup.py install, even in a package manager. A few people (particularly those involved in Arch linux packaging) have been working on minimal PEP 517 build frontend to build wheels and a minimal wheel installer, so possibly early adoption of those technologies will work better for you, @PureTryOut.

[crwilcox]

@PureTryOut did you happen to figure a way around this? If not we may be able to move the version out of setup.py as @busunkim96 previously indicated.

[crwilcox]

Also, this is a workaround another project seems to be using. https://github.com/oppia/oppia/pull/10824/files#diff-b6d8dd401ae816cad17bf9926ce3385dR78

[crwilcox]

Unfortunately I am not reproducing this issue on macos, in a virtualenv, using setup.py build I am passed getting a distribution version when loading the package.

However, as noted above, there is friction using the libraries in GAE. So it is possible we could improve both use cases with the same fix, likely increasing the ability for the version/pkg to be discovered.

❯ python -m venv venv; source venv/bin/activate
❯ python setup.py build
❯ python
Python 3.8.5 (default, Sep 21 2020, 13:30:58)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append("{MY WORKING DIRECTORY}/build/lib")
>>> from google import api_core
>>> import pkg_resources
>>> pkg_resources.get_distribution("google-api-core").version
'1.22.2'

[merwok]

Could you print more info from the object returned by get_distribution?
I don’t think it is finding info in $PWD/build/lib, because the build command does not create egg-info or dist-info directory. I suspect your call is picking up a previous install of google-api-core.

For the original problem: a nice way to handle this is to invert the process, so have the version in the code and use a build tool that knows how to find the version number in the source (flit, setuptools can do that for example). Then you don’t need to depend on a packaging lib at runtime.

[crwilcox]

I think it may have been picking up the dev copy (not the one in build). I can repro if I move to another directory after build :)

❯ python -m venv venv; source venv/bin/activate
❯ pip freeze
❯ python setup.py build
❯ export PYTHONPATH="${PWD}/build/lib"
❯ cd build


❯ python
Python 3.8.5 (default, Sep 21 2020, 13:30:58)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append("/Users/crwilcox/workspace/python-api-core/build/lib")

>>> import pkg_resources
>>> pkg_resources.get_distribution("google-api-core")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/crwilcox/workspace/python-api-core/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 482, in get_distribution
    dist = get_provider(dist)
  File "/Users/crwilcox/workspace/python-api-core/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 358, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/Users/crwilcox/workspace/python-api-core/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 901, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Users/crwilcox/workspace/python-api-core/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 787, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'google-api-core' distribution was not found and is required by the application

However, worth noting, I also can't import the package at all, I can get the error.

❯ python
Python 3.8.5 (default, Sep 21 2020, 13:30:58)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append("/Users/crwilcox/workspace/python-api-core/build/lib")
>>> from google import api_core
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'api_core' from 'google' (unknown location)

[merwok]

Is the package built (copied) into ~/workspace/python-api-core/build/lib?

[crwilcox]

Yes.

❯ pwd
/Users/crwilcox/workspace/python-api-core/build/lib/google
❯ ls
__init__.py __pycache__ api_core

[crwilcox]

I am looking into moving us to not using pkg_resources. Though it is part of namespacing (we do have a fallback to pkgutil)

Also, in order to change this we need to modify our release tooling as it assumes version is specified in setup.py.

[crwilcox]

Not meaning to cross thread the issue, but we do seem to have a repro in AppEngine Standard for Python 2.7

Vendoring google-cloud-datastore into the basic quickstart (hello world) the runtime will fail similarly.

Traceback (most recent call last):
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/4a2384f6b69cef45/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 238, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/4a2384f6b69cef45/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 309, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/4a2384f6b69cef45/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 83, in LoadObject
    obj = __import__(path[0])
  File "/base/data/home/apps/s~crwilcox-test/20201001t151852.429923747926530875/main.py", line 17, in <module>
    from google.cloud import datastore
  File "/base/data/home/apps/s~crwilcox-test/20201001t151852.429923747926530875/lib/google/cloud/datastore/__init__.py", line 62, in <module>
    from google.cloud.datastore.batch import Batch
  File "/base/data/home/apps/s~crwilcox-test/20201001t151852.429923747926530875/lib/google/cloud/datastore/batch.py", line 24, in <module>
    from google.cloud.datastore import helpers
  File "/base/data/home/apps/s~crwilcox-test/20201001t151852.429923747926530875/lib/google/cloud/datastore/helpers.py", line 29, in <module>
    from google.cloud.datastore_v1.proto import datastore_pb2
  File "/base/data/home/apps/s~crwilcox-test/20201001t151852.429923747926530875/lib/google/cloud/datastore_v1/__init__.py", line 17, in <module>
    from google.cloud.datastore_v1 import types
  File "/base/data/home/apps/s~crwilcox-test/20201001t151852.429923747926530875/lib/google/cloud/datastore_v1/types.py", line 25, in <module>
    from google.api_core.protobuf_helpers import get_messages
  File "/base/data/home/apps/s~crwilcox-test/20201001t151852.429923747926530875/lib/google/api_core/__init__.py", line 23, in <module>
    __version__ = get_distribution("google-api-core").version
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/4a2384f6b69cef45/python27/python27_lib/versions/third_party/setuptools-36.6.0/pkg_resources/__init__.py", line 562, in get_distribution
    dist = get_provider(dist)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/4a2384f6b69cef45/python27/python27_lib/versions/third_party/setuptools-36.6.0/pkg_resources/__init__.py", line 436, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/4a2384f6b69cef45/python27/python27_lib/versions/third_party/setuptools-36.6.0/pkg_resources/__init__.py", line 981, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/4a2384f6b69cef45/python27/python27_lib/versions/third_party/setuptools-36.6.0/pkg_resources/__init__.py", line 867, in resolve
    raise DistributionNotFound(req, requirers)
DistributionNotFound: The 'google-api-core' distribution was not found and is required by the application"

[busunkim96]

@crwilcox There's a note in the old google-cloud-python repo that says says the google-cloud-* libs will not work in Py 2 App Engine. https://github.com/googleapis/google-cloud-python#google-cloud-python-client

Heads up! These libraries are supported on App Engine standard's Python 3 runtime but are not supported on App Engine's Python 2 runtime.

Is that use case something we're looking to support?

[crwilcox]

Support is a strong word. That said, seeing that this dependency is causing issues in both areas makes me feel we may want to see if we can avoid it. The interesting thing is that the issue alpine building is the same.

All that in mind, even if I fix this (right now I mocked it out in vendored versions by just saying the version was a static string, we have issues importing rsa in GAE.

"
Traceback (most recent call last):
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/4a2384f6b69cef45/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 238, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/4a2384f6b69cef45/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 309, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/4a2384f6b69cef45/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 83, in LoadObject
    obj = __import__(path[0])
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/main.py", line 17, in <module>
    from google.cloud import datastore
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/google/cloud/datastore/__init__.py", line 62, in <module>
    from google.cloud.datastore.batch import Batch
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/google/cloud/datastore/batch.py", line 24, in <module>
    from google.cloud.datastore import helpers
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/google/cloud/datastore/helpers.py", line 29, in <module>
    from google.cloud.datastore_v1.proto import datastore_pb2
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/google/cloud/datastore_v1/__init__.py", line 18, in <module>
    from google.cloud.datastore_v1.gapic import datastore_client
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/google/cloud/datastore_v1/gapic/datastore_client.py", line 22, in <module>
    from google.oauth2 import service_account
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/google/oauth2/service_account.py", line 77, in <module>
    from google.auth import _service_account_info
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/google/auth/_service_account_info.py", line 22, in <module>
    from google.auth import crypt
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/google/auth/crypt/__init__.py", line 43, in <module>
    from google.auth.crypt import rsa
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/google/auth/crypt/rsa.py", line 27, in <module>
    from google.auth.crypt import _python_rsa
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/google/auth/crypt/_python_rsa.py", line 28, in <module>
    import rsa
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/rsa/__init__.py", line 1
SyntaxError: Non-ASCII character '\xc3' in file /base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/rsa/__init__.py on line 1, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details"

[merwok]

Note that with Python 3, you can have namespace packages without needing pkgutil.

[crwilcox]

Unfortunately this package is last in line to drop Python 2 support as most of our packages depend on it. Moreover, we are seeing this issue, at least on GAE, on Python 2 specifically. I believe due to the way packages are vendored.

[crwilcox]

@PureTryOut @merwok I released https://pypi.org/project/google-api-core/1.22.4/ which should no longer require get_distribution. Please let me know if this works for your use now :)

Unfortunately for many of the other libraries the use is a little more complicated as its use is in both manually authored and generated code. If this approach does work and other libraries (google-cloud-datastore for instance?) would need further work and likely some fixing of generated surfaces to address this there.

[tseaver]

Via #80.