Error: "No module named 'markupsafe'" from py_gapic_library rule

[aohren]

I'm experimenting with the new py_gapic_library rule and get the following error about missing markupsafe module. Do we need it added to requirements.txt?

Traceback (most recent call last):
  File "/usr/local/google/home/aohren/.cache/bazel/_bazel_aohren/585702ed92d2b12f1b55b0ec700de995/sandbox/linux-sandbox/298/execroot/com_google_googleapis/bazel-out/host/bin/external/gapic_generator_python/gapic_plugin.runfiles/gapic_generator_python/gapic/cli/generate.py", line 23, in <module>
    from gapic import generator
  File "/usr/local/google/home/aohren/.cache/bazel/_bazel_aohren/585702ed92d2b12f1b55b0ec700de995/sandbox/linux-sandbox/298/execroot/com_google_googleapis/bazel-out/host/bin/external/gapic_generator_python/gapic_plugin.runfiles/gapic_generator_python/gapic/generator/__init__.py", line 21, in <module>
    from .generator import Generator
  File "/usr/local/google/home/aohren/.cache/bazel/_bazel_aohren/585702ed92d2b12f1b55b0ec700de995/sandbox/linux-sandbox/298/execroot/com_google_googleapis/bazel-out/host/bin/external/gapic_generator_python/gapic_plugin.runfiles/gapic_generator_python/gapic/generator/generator.py", line 15, in <module>
    import jinja2
  File "/usr/local/google/home/aohren/.cache/bazel/_bazel_aohren/585702ed92d2b12f1b55b0ec700de995/sandbox/linux-sandbox/298/execroot/com_google_googleapis/bazel-out/host/bin/external/gapic_generator_python/gapic_plugin.runfiles/gapic_generator_python_pip_deps_pypi__Jinja2_2_11_1/jinja2/__init__.py", line 6, in <module>
    from markupsafe import escape
ModuleNotFoundError: No module named 'markupsafe'
--python_gapic_out: protoc-gen-python_gapic: Plugin failed with status code 1.
Target //google/ads/googleads/v3:googleads_v3_py_gapic failed to build
Use --verbose_failures to see the command lines of failed build steps.

[hkdevandla]

@vam-google, is this something that you are looking at? Please advice.

[vam-google]

@aohren @hkdevandla Yes, please add all missing modules to requirements.txt. This is what I meant that the rules are raw and expectations must be low.

In bazel python rules are "semi-hermetic", unfortunately. What that means is if bazel rules specify explicit dependency on a python module, then exactly this module of exactly this version will be pulled and used by bazel.

But if the module was not specified explicitly, but is installed on the executing machine, the build will pull whatever was installed on the machine.

This makes it hard to detect the full list of dependencies.

@aohren I believe it will be not the only missing module. Please keep installing them and keep notest of what you had to install, once it stops complaining about missing modules.

[hkdevandla]

Sounds good. Thanks @vam-google.

@aohren , please let us know if you have questions.

[aohren]

@vam-google Understood, I did try locally installing the markupsafe package on my build machine, but that didn't seem to resolve the error. I can keep digging into it, but would there be anything else required to get the Bazel to see the module installed on the executing machine?

[aohren]

I've also tried cloning gapic-generator-python project locally, pointing my WORKSPACE to it via local_repository instead of GitHub, and adding markupsafe==1.1.1 to the requirements.txt. Still getting the ModuleNotFoundError error.

[vam-google]

@aohren Sorry, I forgot to mention that each dependency must be added in two places:
requirements.txt and BUILD.bazel file to the list of the py_binary target dependencies.

On my machine the dependency is called 'MarkupSafe' (capital M and S), so it should be something like MarkupSafe == 1.1.1 in requirements.txt
requirement("MarkupSafe") in BUIDL.bazel (in the root of this repo).

When you install modules on your machine's python, please make sure that they are installed for the proper version of python (the one used by bazel, which is most probably the latest version of Python on your machine).

Please let me know if that works for you (I was able to reproduce the issue by removing the module from my machine and then "fix" it by putting the module in the two files).

[vam-google]

@aohren Hi, Adam, did you have a chance to check if the suggested solution works for you?

[aohren]

Ah yes, thanks. Adding the analogous line in BUILD.bazel fixed the issue.

One thing to note in general is that Fetching @gapic_generator_python_pip_deps step takes a little >5 mins from a clean build. Makes debugging the dependencies a long process.

[vam-google]

Yes, I noticed that too. It is not always the case (sometimes it is fast). It is weird and should not take that much time. This dependency pulling is done by rules_python library we depend on (collection of generic python rules). I honestly don't know what is going on here =(. We need to ask the python rules maintainers.

[aohren]

From what I can tell, when it's stuck waiting for the ~5 mins, there are many compiler and assembler processes running in the background (e.g., cc1plus, as, etc.). My guess is that it's rebuilding at least some of the dependencies (python runtime, pip, dependencies, etc.) from source on the local machine and it's taking a huge amount of time.

I can reproduce this consistently by doing a bazel clean --expunge before building.

[arithmetic1728]

@aohren Just released 0.21.1 with your PR. Closing the issue.

[arithmetic1728]

It seems only one of the issues mentioned is solved, reopening the issue.

[aohren]

Created issue #400 to address the second concern, as it's a separate issue. We can close this item out.