Cannot import other package under google namespace if install protobuf

[fengli79]

I get the same issue (#1296 (comment)) when using protobuf 3.5.1.
The protobuf is installed in:

site-packages/
google/
protobuf/
init.py
There's no "init.py" install under the google package, and I have to import the google.protobuf package to make other packages under the google namespace be available.

[fengli79]

To reproduce:

1. Use virtualenv to create a new environment.
2. pip install grpcio
3. create a dir named "src"
4. create a dir named "google" under "src" dir.
5. create an __init__.py file under "google" dir, with content:

try:
  __import__('pkg_resources').declare_namespace(__name__)
except ImportError:
  __path__ = __import__('pkgutil').extend_path(__path__, __name__)

6. create a test.py inside the "google" dir, with content:
   print("hello")
7. create another test.py under ""src" dir, with content:

  import google.protobuf
  import google.test

8. Add the "src" dir to the PYTHONPATH environment variable.
9. under "src" dir, run:
   python test.py

It should print out "hello".
If you remove the "import google.protobuf" statement, it will report the test module cannot be found.

[craigcitro]

I don't think this is anything particular to google.protobuf: I can produce precisely the same behavior as the above with any of the existing google packages (eg google-auth or google-cloud).

I believe that the issue we're hitting is the Note: in the namespace packages docs. Adding @theacodes, since she's wiser about these sorts of things: Thea, should we try and stage a change over to using pkgutil-based installation for the various Google packages?

[theacodes]

Having conflicting entries in PYTHONPATH causes problems with everything, and especially namespace packages. Using pkgutil does not fix this (and it's totally infeasible for us to switch in a way that doesn't absolutely break the universe).

My advice: Don't ever touch PYTHONPATH, and if you must, you can't stick google folder in there and not expect to to clobber the import machinery. Even if google weren't a namespace package, that would clobber whatever is in site-packages.

@fengli79 why are you creating a google folder anyway? Are you developing a package that needs to live in the google namespace? If so, we need to chat.

[arthur-tacca]

Note that when you run a Python script from the command line, Python adds the directory containing that script to the import path list. (This means that if you run "foo/bar.py", then foo is added to the Python path, NOT the current working directory, which is a common source of confusion.) In the situation described in the original post, src/ is included in the path list for both that reason and again, a second time, from being in the PYTHONPATH. You can check this by including import sys; print(sys.path). I doubt this is the cause of your overall problem, but it doesn't look good.

@jonparrott Setting PYTHONPATH only adds entries to your sys.path, not clobber them, and certainly doesn't clobber site-packages. (Obviously it does clobber anything that was previously in PYTHONPATH if it was already set and you don't use it in the new value.)

I think the real problem is that the original post is trying to use google as both a namespace package (can only contain other packages, but they can be anywhere) and a regular package (directly contains a module, namely google.test). If you change the original example so that test.py is in a subdirectory src/google/foo, add src/google/foo/__init__.py, and try import google.foo or import google.foo.test, everything should work OK (as well as namespace packages ever do).

[theacodes]

Obviously it does clobber anything that was previously in PYTHONPATH if it was already set and you don't use it in the new value.

This is what I meant, but it's actually a little more insidious that than. Installed namespace packages typically create a .pth file in site-packages that setups up the {module}.__path__ in sys.modules so that the disparate namespace packages can be found. Having a folder named {module} in the current directory or in PYTHONPATH causes this machinery to break.

If you change the original example so that test.py is in a subdirectory src/google/foo, add src/google/foo/init.py, and try import google.foo or import google.foo.test, everything should work OK (as well as namespace packages ever do).

Not necessarily. The real solution is to install the test package using pip install -e . (do not use python setup.py develop).

[rodrigoaliste]

Any updates?

[Flamefire]

Related from the docs: http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html#the-init-py-trap

In short: Do not use an __init__.py in the google folder and use Python >= 3.3

[SikandAlex]

Yo Google this is seriously ridiculous. People have been having this issue for going on 5 years. It shouldn't take me hours of work to try to fix this. All I want is to be able to use the google-cloud-vision Python package and no matter what I do I can't avoid protobuf not being properly recognized.

When are you going to address this?

Ubuntu 20.04
Python 3.8.3

[kriswuollett]

I'm also having issues with this along with https://github.com/googleapis/googleapis. Originally I filed bazelbuild/rules_python#577, but looks like the issue is here? I created an example workspace that can be used to reproduce the issue.

I'm using:

bazel 4.2.1
macOS 12.0.1 (Intel)
Python 3.9.8 (main, Nov 10 2021, 09:21:22) 
[Clang 13.0.0 (clang-1300.0.29.3)]

[mattoberle]

Sorry to bump an old issue, but I figured I'd share my experience with this problem for other Bazel users.

The key to avoiding this problem is --incompatible_default_to_explicit_init_py.

A related issue was raised in #4658 and a change was merged which fixed the problem.
Later, that change was removed by this commit bf0c69e, which re-introduced the problem.

Without the --incompatible_default_to_explicit_init_py flag Bazel will create empty __init__.py files in each directory that doesn't have one.
The empty __init__.py is what breaks the google.* namespace.
Including google/__init__.py or omitting __init__.py solves the problem.

I was actually in the process of putting together a PR to add google/__init__.py back to the glob (because that would work with and without the flag)... but it seems like it was removed intentionally.

In summary, Bazel users have to opt-in to not break the google.* package namespace when using protobuf.
I'd be happy to open the PR I put together if that's not the intended behavior.

[MrGuoRanDuo]

fuck u google

[bfsoyc]

Sorry to bump an old issue, but I figured I'd share my experience with this problem for other Bazel users.

The key to avoiding this problem is --incompatible_default_to_explicit_init_py.

A related issue was raised in #4658 and a change was merged which fixed the problem. Later, that change was removed by this commit bf0c69e, which re-introduced the problem.

Without the --incompatible_default_to_explicit_init_py flag Bazel will create empty __init__.py files in each directory that doesn't have one. The empty __init__.py is what breaks the google.* namespace. Including google/__init__.py or omitting __init__.py solves the problem.

I was actually in the process of putting together a PR to add google/__init__.py back to the glob (because that would work with and without the flag)... but it seems like it was removed intentionally.

In summary, Bazel users have to opt-in to not break the google.* package namespace when using protobuf. I'd be happy to open the PR I put together if that's not the intended behavior.

Thanks for your sharing.
Any update on this?
I take a look into the google/__init__.py, which seems to handle the namespace nicely.
Is there other concerns?
Btw, I know how to patch the http_archive rule, however I didn't figure out how to patch pip_parse rule yet. It's unfriendly for bazel user.

[github-actions]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment.

This issue is labeled inactive because the last activity was over 90 days ago.

[github-actions]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please reopen it.

This issue was closed and archived because there has been no new activity in the 14 days since the inactive label was added.