pants 0.0.32 cannot build/run python3 binaries #1658

jsirois · 2015-06-10T00:11:11Z

As initially reported by Grant here: https://groups.google.com/d/topic/pants-devel/dr_vUO_gG_4/discussion

On Tue, Jun 9, 2015 at 5:43 PM, Grant Wu [email protected] wrote:
Hello!

Grant here. Member of a team of avid Pants-wearers. Running into an issue for which I've failed to find a solution.

I have Python code that uses 'raise...from' syntax as outlined here. And here is a Gist containing some test code I'm trying to run.

A file named "test.py" containing only the contents of the above Gist fails to run / bundle with Pants. Error:

"Exception message: invalid syntax (test.py, line 4)"

The same test.py runs successfully when invoked with Python3.4.3, and gives the expected syntax error with Python2.

Here is my pants.ini. I'm specifying "interpreter_requirement: CPython>=3.4,<3.5".

I'm specifying a version of Python that should support 'raise...from', so I can't figure out why Pants is giving me a syntax error. I'm wondering if there's something else I need to do to tell Pants which version of Python I'm using.

Am I missing something? Thanks so much in advance for your help.

I did a minimal repo like so:

Setup a pants python repo:

jsirois@gill ~ $ mkdir /tmp/test
jsirois@gill ~ $ cd /tmp/test
jsirois@gill /tmp/test $ virtualenv-2.7 build-support/pants.venv
...
jsirois@gill /tmp/test $ source build-support/pants.venv/bin/activate
(pants.venv)jsirois@gill /tmp/test $ pip install pantsbuild.pants
...
(pants.venv)jsirois@gill /tmp/test $ cat << EOF > pants.ini
> [python-setup]
> interpreter_requirement: CPython>=3.4,<3.5
> EOF

Setup Grant's case:

(pants.venv)jsirois@gill /tmp/test $ cat << EOF > main.py
> try:
>     print(1 / 0)
> except Exception as exc:
>     raise RuntimeError("Something bad happened") from exc
> EOF
(pants.venv)jsirois@gill /tmp/test $ cat << EOF > BUILD
> python_binary(
>   name='main',
>   source='main.py'
> )
> EOF

Repro the run ... not quite so fast

(pants.venv)jsirois@gill /tmp/test $ pants run :main
...
17:50:25 00:00   [resolve]
17:50:25 00:00     [ivy]
                   Failed to resolve target for tool: //:nailgun-server.
                   This target was obtained from option nailgun-server in scope resolve.ivy.
                   You probably need to add this target to your tools BUILD file(s), usually located in the workspace root.

FAILURE
...

URG - tracked here: Pants requires ivy for a python-only project #940

Add the nailgun tool dep:

(pants.venv)^jsirois@gill /tmp/test $ cat << EOF > BUILD.tools
> jar_library(
>   name='nailgun-server',
>   jars=[
>     jar('com.martiansoftware', 'nailgun-server', '0.9.1')
>   ]
> )
> EOF

5.) Really do the repro:

(pants.venv)jsirois@gill /tmp/test $ pants binary :main --print-exception-stacktrace
...
17:52:22 00:00   [binary]
17:52:22 00:00     [python-binary-create]
                   Detected interpreter /home/jsirois/.pyenv/versions/3.4.2/bin/python3.4: CPython-3.4.2
                   Detected interpreter /usr/bin/python3.4: CPython-3.4.3fatal: Not a git repository (or any parent up to mount point /tmp)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

17:52:23 00:01       [chroot]
               FAILURE

Exception caught:
  File "/tmp/test/build-support/pants.venv/bin/pants", line 9, in <module>
    load_entry_point('pantsbuild.pants==0.0.32', 'console_scripts', 'pants')()
  File "/tmp/test/build-support/pants.venv/lib/python2.7/site-packages/pants/bin/pants_exe.py", line 81, in main
    _run(exiter)
  File "/tmp/test/build-support/pants.venv/lib/python2.7/site-packages/pants/bin/pants_exe.py", line 74, in _run
    result = goal_runner.run()
  File "/tmp/test/build-support/pants.venv/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 174, in run
    result = self._do_run()
  File "/tmp/test/build-support/pants.venv/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 230, in _do_run
    return engine.execute(context, self.goals)
  File "/tmp/test/build-support/pants.venv/lib/python2.7/site-packages/pants/engine/engine.py", line 26, in execute
    self.attempt(context, goals)
  File "/tmp/test/build-support/pants.venv/lib/python2.7/site-packages/pants/engine/round_engine.py", line 212, in attempt
    goal_executor.attempt(explain)
  File "/tmp/test/build-support/pants.venv/lib/python2.7/site-packages/pants/engine/round_engine.py", line 45, in attempt
    task.execute()
  File "/tmp/test/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/tasks/python_binary_create.py", line 38, in execute
    self.create_binary(binary)
  File "/tmp/test/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/tasks/python_binary_create.py", line 51, in create_binary
    with self.temporary_chroot(interpreter=interpreter, pex_info=pexinfo, targets=[binary], platforms=binary.platforms) as chroot:
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/tmp/test/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/tasks/python_task.py", line 104, in temporary_chroot
    chroot.dump()
  File "/tmp/test/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/python_chroot.py", line 178, in dump
    self._dump_library(lib)
  File "/tmp/test/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/python_chroot.py", line 112, in _dump_library
    copy_to_chroot(library.target_base, relpath, self._builder.add_source)
  File "/tmp/test/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/python_chroot.py", line 107, in copy_to_chroot
    add_function(src, path)
  File "/tmp/test/build-support/pants.venv/lib/python2.7/site-packages/pex/pex_builder.py", line 136, in add_source
    pyc_object = CodeMarshaller.from_py(fp.read(), env_filename)
  File "/tmp/test/build-support/pants.venv/lib/python2.7/site-packages/pex/marshaller.py", line 70, in from_py
    code = compile(py.replace('\r\n', '\n').replace('\r', '\n'), filename, 'exec')

Exception message: invalid syntax (main.py, line 4)

jsirois · 2015-06-10T14:27:13Z

This requires a fix in pex which requires upgrading to the pex 1.0.0 series. I'm tackling the upgrade 1st in #1440

gw · 2015-06-10T16:48:36Z

Many thanks, John. Really appreciate your time on this. Let us know if there's anything we can do to help!

jsirois · 2015-06-12T04:39:42Z

Working the underlying issue in pex here:

jsirois · 2015-06-19T16:09:11Z

Noting that pex RB https://rbcommons.com/s/twitter/r/2350/ is approved but pants still needs pex RB https://rbcommons.com/s/twitter/r/2341/ before it can upgrade to the new pex to solve this issue. I haven't forgotten about you over here.

gw · 2015-06-22T20:42:58Z

Thanks so much for the updates, John!

jsirois · 2015-07-03T19:03:06Z

The fix for pex resolves in https://rbcommons.com/s/twitter/r/2341/ is now in master so pants can consume the #127 compiler fix that will get this issue resolved with the next pex release.

jsirois · 2015-07-07T02:26:11Z

I just confirmed that my repro case above now work using a pantsbuild.pants sdist locally released from https://rbcommons.com/s/twitter/r/2438/:

$ ./dist/main.pex 
Traceback (most recent call last):
  File "main.py", line 2, in <module>
    print(1 / 0)
ZeroDivisionError: division by zero

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".bootstrap/_pex/pex.py", line 319, in execute
  File ".bootstrap/_pex/pex.py", line 254, in _wrap_coverage
  File ".bootstrap/_pex/pex.py", line 286, in _wrap_profiling
  File ".bootstrap/_pex/pex.py", line 362, in _execute
  File ".bootstrap/_pex/pex.py", line 420, in execute_entry
  File ".bootstrap/_pex/pex.py", line 425, in execute_module
  File "/usr/lib/python3.4/runpy.py", line 185, in run_module
    return _run_code(code, {}, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.4/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "main.py", line 4, in <module>
    raise RuntimeError("Something bad happened") from exc
RuntimeError: Something bad happened

The https://rbcommons.com/s/twitter/r/2438/ change (upgrade to pex 1.0.1) will land before Friday and we now have regular pants releases every Friday, so this should all work for you by EOW.

This is a minimal upgrade that does not engage any of the new pex features, but it does pull in a bugfix necessary to squash: #1658 Testing Done: CI went green here: https://travis-ci.org/pantsbuild/pants/builds/69824504 Bugs closed: 1440, 1658, 1761 Reviewed at https://rbcommons.com/s/twitter/r/2438/

jsirois · 2015-07-07T17:59:12Z

Fix in master @ 3287103

This should go out in the 0.0.35 release on Friday, 7/10/15

gw · 2015-07-14T22:52:06Z

Much gratitude, John! We'll upgrade to 0.0.35 and take it for a spin :)

The team really appreciates your time!

👍

Allan-Nava · 2018-03-27T12:38:31Z

It works with python3 ?

stuhood · 2018-03-27T17:46:01Z

@Allan-Nava : Yep! Twitter is currently rolling out python3 internally, so it's a good time to give it a try.

Allan-Nava · 2018-03-27T18:34:51Z

Perfect! But is working with django framework? 2018-03-27 19:46 GMT+02:00 Stu Hood <[email protected]>:

…

@Allan-Nava <https://github.com/Allan-Nava> : Yep! Twitter is currently rolling out python3 internally, so it's a good time to give it a try. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1658 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AVdMg5DpOmQJU_6TCkFdVBbTv-npvKWmks5tinrbgaJpZM4E9zMC> .

stuhood · 2018-03-27T18:51:15Z

I do not know. That would be a separate ticket.

Allan-Nava · 2018-03-27T18:52:54Z

@stuhood I used django 1.11.5 with python3

jsirois added the bug label Jun 10, 2015

jsirois added the in-progress label Jun 10, 2015

dargueta-work mentioned this issue Jun 10, 2015

Pants permanently crashes, even uninstall/reinstall can't fix it #1661

Closed

jsirois added in-progress and removed in-progress labels Jun 10, 2015

jsirois mentioned this issue Jun 12, 2015

Bytecode compile using the correct interpreter. pex-tool/pex#126

Closed

jsirois mentioned this issue Jun 12, 2015

PEXBuilder does not respect interpreter when compiling source to bytecode pex-tool/pex#127

Closed

jsirois mentioned this issue Jul 7, 2015

Upgrade to pex 1.0.1 #1761

Closed

jsirois closed this as completed Jul 7, 2015

jsirois removed the in-progress label Jul 7, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pants 0.0.32 cannot build/run python3 binaries #1658

pants 0.0.32 cannot build/run python3 binaries #1658

jsirois commented Jun 10, 2015

jsirois commented Jun 10, 2015

gw commented Jun 10, 2015

jsirois commented Jun 12, 2015

jsirois commented Jun 19, 2015

gw commented Jun 22, 2015

jsirois commented Jul 3, 2015

jsirois commented Jul 7, 2015

jsirois commented Jul 7, 2015

gw commented Jul 14, 2015

Allan-Nava commented Mar 27, 2018

stuhood commented Mar 27, 2018

Allan-Nava commented Mar 27, 2018 via email

stuhood commented Mar 27, 2018

Allan-Nava commented Mar 27, 2018

pants 0.0.32 cannot build/run python3 binaries #1658

pants 0.0.32 cannot build/run python3 binaries #1658

Comments

jsirois commented Jun 10, 2015

jsirois commented Jun 10, 2015

gw commented Jun 10, 2015

jsirois commented Jun 12, 2015

jsirois commented Jun 19, 2015

gw commented Jun 22, 2015

jsirois commented Jul 3, 2015

jsirois commented Jul 7, 2015

jsirois commented Jul 7, 2015

gw commented Jul 14, 2015

Allan-Nava commented Mar 27, 2018

stuhood commented Mar 27, 2018

Allan-Nava commented Mar 27, 2018 via email

stuhood commented Mar 27, 2018

Allan-Nava commented Mar 27, 2018