Split out pex_rules.bzl to a separate project?

[benley]

The pex python rules found in //tools/rules:pex_rules.bzl would be useful to numerous projects beyond Heron. I have started maintaining an enhanced variant of those rules in a new repository: https://github.com/benley/bazel_rules_pex.

Some of the changes I made:

• Everything works with bazel's sandboxing enabled now(!)
• setuptools and wheel are fetched as external http_file resources, not vendored into the project tree.
• The pex wrapper can now correctly (but optionally) use .whl dependencies along with eggs.
• The rules now work when loaded from a remote skylark repository.
• There is full documentation, generated from the source docstrings via skydoc.
• Added an entrypoint attribute to pex_binary, allowing a python module name to be used instead of a main file. Useful for building pexes for external egg/whl dependencies.
• pex_library targets can now be passed to the native py_library, py_binary, and py_test rules as a dependency, and vice versa.
• main module resolution now works correctly for generated source files.
• pex_test no longer invokes the py.test runner by default. Instead it works more like bazel's built-in py_test rule. The previous py.test behaviour is now available as a pex_pytest macro. (incompatible change! but trivially reconciled)
• pex_test is now a trivial special case of pex_binary.
• pex_pytest itself is now a fairly simple wrapper around pex_binary.
• pex_pytest now automatically adds a dependency on py.test
• The resources attribute has been renamed to data to be consistent with most other Bazel rule types. Its behaviour has not changed, just the name. (incompatible change! but trivially reconciled.)
• There is a lot less duplicated code.

Would there be any interest in combining efforts here? My goal is to get rules_pex published as a semi-official project in the bazelbuild github organization, alongside the various other language-specific Bazel rules. I would also be happy to contribute some or all of these improvements back to Heron, whether or not Heron ends up importing the rules from a separate project.

Thanks! Looking forward to any feedback.

[billonahill]

This is great @benley, we'd love to have the bazel/pex stuff pulled out of Heron and put closer to the bazel repo. If you want to submit a PR to remove all of this pex code and instead reference it from a remote source somehow, we'd be glad to accept that.

The only benefit we get from having it in the repo currently is that when things go wrong with the pex build, there's sometimes little helpful logging to pinpoint the issue. In that case we manually change tracer.py's should_log to return True. This is an ugly hack and it would be much better if bazels verbose flags caused verbosity to be correctly set within pex. This is something we've been meaning to fix for a while.

[benley]

Cool, I'm glad to hear that :-)

I don't know if there's currently a way to hook into bazel's verbosity options from within skylark, but lacking that it ought to be easy enough to add a debug mode toggle to the pex rules.

[kramasamy]

@benley - this is really awesome. As @billonahill mentioned, it will be great have a PR and a debug option. It will be nice if you could integrate these rules permanently into Bazel. Currently, Bazel lacks the ability to build self contained python binaries.

[kramasamy]

Another suggestion would be to upgrade to new version of PEX? PEX has new versions. If we could use the repository of PEX for code https://github.com/pantsbuild/pex based on a particular release and your rules using git repo rules in bazel WORKSPACE, we can seamlessly upgrade PEX as new versions are being released.

[benley]

I would love to get these rules using an up-to-date upstream version of PEX without any modifications, but that is currently blocked on at least one bug: pex-tool/pex#277

[kramasamy]

@benley - let us see if we could resolve this bug.

[benley]

Relatedly: is there still a need to support python2.4 in the Heron codebase? I'm now running into some cruft that could just go away if backwards compatibility all the way to 2.4 isn't required.

[kramasamy]

@benley - we are in the process of upgrading the pex. Please check out the pull request #1052

[benley]

I'll try to keep my fork in sync with the update! Haven't had time to do much work on this in the last week but I'll get back to it soon.

[kramasamy]

@benley - with PEX upgraded, it might be good to get pex rules upgraded. Let us know when you are ready.

[benley]

I've got my copy of pex upgraded now. I want to do a little more testing before diving in with heron; stay tuned.

[kramasamy]

Thanks. Look forward to it.

Sent from my iPhone

On Jul 14, 2016, at 11:47 AM, Benjamin Staffin [email protected] wrote:

I've got my copy of pex upgraded now. I want to do a little more testing before diving in with heron; stay tuned.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[kramasamy]

@benley - any update on this project? Let us know - we will be happy to help if you need anything.