Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bazel not picking up the right protobuf python library. #1209

Closed
vinuraja opened this issue Apr 28, 2016 · 6 comments
Closed

Bazel not picking up the right protobuf python library. #1209

vinuraja opened this issue Apr 28, 2016 · 6 comments
Assignees
@davidzchen
Labels
P2 We'll consider working on this in future. (Assignee optional) type: bug

Comments

@vinuraja
Copy link

Trying to build TensorFlow Serving (specific rule //tensorflow_serving/session_bundle/example:half_plus_two) fails with this error.

Protobufs are imported as the submodule of TensorFlow, which is itself a submodule of TensorFlow Serving.

It seems like instead of getting this protobuf python library its picking up my local python protobuf library resulting in this error.

ERROR: /tmp/staging/serving-staging/tensorflow_serving/session_bundle/example/BUILD:34:1: Executing genrule //tensorflow_serving/session_bundle/example:half_plus_two failed: bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
Traceback (most recent call last):
  File "/usr/local/google/home/vinuraja/.cache/bazel/_bazel_vinuraja/4179d7df99792da64adb58f91ec06044/serving-staging/bazel-out/host/bin/tensorflow_serving/session_bundle/example/export_half_plus_two.runfiles/tensorflow_serving/session_bundle/example/export_half_plus_two.py", line 33, in <module>
    import tensorflow as tf
  File "/usr/local/google/home/vinuraja/.cache/bazel/_bazel_vinuraja/4179d7df99792da64adb58f91ec06044/serving-staging/bazel-out/host/bin/tensorflow_serving/session_bundle/example/export_half_plus_two.runfiles/external/tf/tensorflow/__init__.py", line 23, in <module>
    from tensorflow.python import *
  File "/usr/local/google/home/vinuraja/.cache/bazel/_bazel_vinuraja/4179d7df99792da64adb58f91ec06044/serving-staging/bazel-out/host/bin/tensorflow_serving/session_bundle/example/export_half_plus_two.runfiles/external/tf/tensorflow/python/__init__.py", line 49, in <module>
    from tensorflow.core.framework.graph_pb2 import *
  File "/usr/local/google/home/vinuraja/.cache/bazel/_bazel_vinuraja/4179d7df99792da64adb58f91ec06044/serving-staging/bazel-out/host/bin/tensorflow_serving/session_bundle/example/export_half_plus_two.runfiles/external/tf/tensorflow/core/framework/graph_pb2.py", line 9, in <module>
    from google.protobuf import symbol_database as _symbol_database
  File "/usr/local/google/home/vinuraja/.cache/bazel/_bazel_vinuraja/4179d7df99792da64adb58f91ec06044/serving-staging/bazel-out/host/bin/tensorflow_serving/session_bundle/example/export_half_plus_two.runfiles/external/tf/google/protobuf/symbol_database.py", line 180, in <module>
    _DEFAULT = SymbolDatabase(pool=descriptor_pool.Default())
AttributeError: 'module' object has no attribute 'Default'
Target //tensorflow_serving/session_bundle/example:half_plus_two failed to build

Output of 'bazel info'

bazel-bin: /usr/local/google/home/vinuraja/.cache/bazel/_bazel_vinuraja/4179d7df99792da64adb58f91ec06044/serving-staging/bazel-out/local_linux-fastbuild/bin
bazel-genfiles: /usr/local/google/home/vinuraja/.cache/bazel/_bazel_vinuraja/4179d7df99792da64adb58f91ec06044/serving-staging/bazel-out/local_linux-fastbuild/genfiles
bazel-testlogs: /usr/local/google/home/vinuraja/.cache/bazel/_bazel_vinuraja/4179d7df99792da64adb58f91ec06044/serving-staging/bazel-out/local_linux-fastbuild/testlogs
command_log: /usr/local/google/home/vinuraja/.cache/bazel/_bazel_vinuraja/4179d7df99792da64adb58f91ec06044/command.log
committed-heap-size: 508MB
execution_root: /usr/local/google/home/vinuraja/.cache/bazel/_bazel_vinuraja/4179d7df99792da64adb58f91ec06044/serving-staging
gc-count: 78
gc-time: 1065ms
install_base: /usr/local/google/home/vinuraja/.cache/bazel/_bazel_vinuraja/install/46732835f8fdfb418a51e9e1954eb778
max-heap-size: 8363MB
message_log: /usr/local/google/home/vinuraja/.cache/bazel/_bazel_vinuraja/4179d7df99792da64adb58f91ec06044/message.log
output_base: /usr/local/google/home/vinuraja/.cache/bazel/_bazel_vinuraja/4179d7df99792da64adb58f91ec06044
output_path: /usr/local/google/home/vinuraja/.cache/bazel/_bazel_vinuraja/4179d7df99792da64adb58f91ec06044/serving-staging/bazel-out
package_path: %workspace%
release: release 0.2.1
server_pid: 1379
used-heap-size: 85MB
workspace: /tmp/staging/serving-staging
@meteorcloudy meteorcloudy added type: bug P2 We'll consider working on this in future. (Assignee optional) labels Apr 29, 2016
@meteorcloudy
Copy link
Member

@damienmg Can you look into this?

@damienmg
Copy link
Contributor 

$ bazel help build | grep python
..............
  --force_python (py2, py3, py2and3, py2only or py3only; default: see description)
  --host_force_python (py2, py3, py2and3, py2only or py3only; default: see description)
  --python2_path (a string; default: "python")
  --python3_path (a string; default: "python3")

Maybe try to setting the path to python binary. It should definitely have protobuf in the python path. @davidzchen might have more insights.

@vinuraja
Copy link
Author

@damienmg

Well, things are working for me after I installed the latest python protobuf library. But Bazel shouldn't be using my local protobuf library, but the ones in the submodule.

@damienmg
Copy link
Contributor

damienmg commented May 2, 2016

Totally agree, that's definitely should not happen.

@davidzchen davidzchen mentioned this issue May 3, 2016
Closed
@davidzchen davidzchen mentioned this issue May 12, 2016
Closed
@davidzchen
Copy link
Member

This is blocking tensorflow/tensorflow#1289. I'll go ahead and prioritize this and investigate.

@davidzchen davidzchen self-assigned this May 12, 2016
@davidzchen davidzchen mentioned this issue May 21, 2016
Merged
davidzchen added a commit to davidzchen/protobuf that referenced this issue May 26, 2016
@davidzchen
Bazel build: Keep generated sources and Python runtime in the same di…
02cd45c 
…rectory.

Users often encounter a Python import error when trying to build Python
protos if protobuf is installed locally on the machine. In this case,
Python ends up looking in the wrong directory when importing files (see
bazelbuild/bazel#1209 and tensorflow/tensorflow#2021). It seems that the
problem is caused by Python getting confused when there are Python
source files that are meant to be part of the same package but are
in separate directories.

Prior to protocolbuffers#1233, the Bazel build setup would copy the Python
runtime sources and all generated sources for the builtin protos into
the root directory (assuming that the protobuf tree is vendored in a
google/protobuf directory).

With protocolbuffers#1233, the two sets of sources are kept in their respective
directories but both `src/` and `python/` are added to the `PYTHONPATH`
using the new `imports` attribute of the Bazel Python rules. However,
both the runtime sources and the generated sources are under the same
package: `google.protobuf`, causing Python to become confused when
trying to import modules that are in the other directory.

This patch adds a workaround to the Bazel build to add a modified
version of the original `internal_copied_filegroup` macro to copy the
`.proto` files under `src/` to `python/` before building the
`py_proto_library` targets for the builtin protos. This ensures that the
generated sources for the builtin protos will be in the same directory
as the corresponding runtime sources.

This patch was tested with the following:
* All Python tests in protobuf
* All Python tests in tensorflow
* All tests in [Skydoc](https://github.com/bazelbuild/skydoc)
* Importing protobuf as `//google/protobuf`
* Importing and binding targets under `//external`
* Importing protobuf as `//third_party/protobuf`
@davidzchen
Copy link
Member

This turned out to be a problem in the protobuf Bazel build setup. protocolbuffers/protobuf#1586 has been merged to fix this problem. Once tensorflow/tensorflow#1289 is merged to bump TensorFlow's dependency to the protobuf version that contains the fix, this problem will be fixed on the TensorFlow side as well.

bazel-io pushed a commit to bazelbuild/skydoc that referenced this issue Jun 15, 2016
@davidzchen
Bump Skylark's protobuf dependency to ed87c1f.
2cb63fd 
This way, we have the fix for the Python protobuf build issue
(see bazelbuild/bazel#1209).

--
MOS_MIGRATED_REVID=124878800
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@davidzchen davidzchen

Labels
P2 We'll consider working on this in future. (Assignee optional) type: bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@damienmg @meteorcloudy @davidzchen @vinuraja