New python-implementation? Package pyston

[h-vetinari]

Hot off the press, v2 of pyston, claims to be 20% faster than python with lower memory footprint than PyPy.

Thoughts @isuruf @conda-forge/core?

[jakirkham]

Timely! 😄 Was just reading that blogpost earlier. Yeah this seems like a good idea. We might want to confirm, which Python versions they are targeting. Historically this was Python 2, but the blogpost seems to indicate Python 3 support (though unclear as to which one(s)). Also the blogpost seems to indicate this hasn't been open sourced yet, but will be soon (feel free to correct me if I misread). So maybe this buys us some time to figure out how we want to do this when it is open sourced.

[h-vetinari]

Pyston 2.2 has now been open-sourced 🙃
https://github.com/pyston/pyston

[wolfv]

Yeah, this is exciting! the 2x llvm builds in there scare me a bit, though :)

[chrisburr]

the 2x llvm builds in there scare me a bit, though :)

Yup! At a glance it looks like:

• one is a patched LLVM for https://github.com/facebookincubator/BOLT
• pyston itself seems to depend on LLVM 10 so it might not be too bad (the commit is between 10.0.0 and 10.0.1)

[h-vetinari]

The good thing is that BOLT will likely become part of LLVM eventually (though that'll still take month or quarters, realistically).

[h-vetinari]

Here's a tracking issue for LLVM integration of BOLT: facebookarchive/BOLT#183

[jakirkham]

Looks like Anaconda hired the devs working on Pyston. More details in this Pyston blogpost

[jakirkham]

Coming to this a bit late, but it appears pyston has included some packages in their own channel. So maybe that is a simple way to install them and get started

[h-vetinari]

Thanks John! I think it might be worth considering eventually(?) just providing that directly from conda-forge, then also those resolver errors mentioned in the blog post would be much less of an issue (because there'd at least be a migrator to track).

Ping @kmod for visibility (was just conversing with them in pyston/pyston#180).

[jakirkham]

No objection to that. Though there is always a balance to be struck in terms of what is most productive/effective for everyone. If rebuilds are quite frequent, that may mean just using the pyston channel. Whenever things settled down, migrating to conda-forge may make sense. Would be good to hear from Kevin, et al. on what state things are in and when that transition may make sense 🙂

[kmod]

We'd love to be included! We built quite a few of the top conda-forge packages for our channel, and the main issues were not Pyston-related (we had to deal with a couple circular dependencies, as well as some feedstocks that don't build currently because new versions of dependencies broke them). We do also have a small amount of patches for Pyston compatibility, and we haven't submitted them yet to the projects because we are waiting on some discussions with the Python packaging community about how they would like to support alternative implementations.

Here's the script that I wrote to build the packages, which also includes most of the patches that we applied to the feedstocks:
https://github.com/pyston/pyston/blob/pyston_main/pyston/conda/build_feedstock.py

How do you all deal with building packages for a new version of CPython? I guess I should have asked that before building these packages ourselves. Hopefully adding Pyston isn't too much harder than that.

We don't anticipate breaking ABI compatibility (causing rebuilds) very often, we're currently hoping that we only have to do that when we retarget to a new Python version.

[jakirkham]

Just to confirm, packages using Pyston need to be rebuilt against it or is there some level of ABI/API compatibility with CPython?

Also curious to get your thoughts on whether C API standard projects like HPy, abi3, etc. might be used in Pyston (admittedly this is likely a longer term thing)?

[kmod]

We are API compatible but not ABI compatible. (We support the stable ABI but don't yet properly flag that.) We rebuilt all the binary packages but didn't need to do anything with the pure-python packages.

We're in an interesting position where we support compiling any package from source, but some packages (like tensorflow, numba, pytorch) are essentially impossible for a user to build on their machine. So we're hoping to get to a point where we have Pyston-native extensions available so people can have the best performance, but as an intermediate step it'd be great to have more stable-abi or HPy-universal-packages!

[jakirkham]

So we do have the python_abi for tracking different Python versions & implementations.

Also we do run migrations for new Python versions and PyPy. Those are currently done globally. Though we have done this in an opt-in basis for some things like ARM & PPC.

Adding a new Python implementation is doable. We might want to discuss how many packages we want to rebuild (IOW do we have a specific list or are we trying to do everything?).

cc @conda-forge/core (guessing others will have thoughts on these and have additional questions as well)

[beckermr]

We didn't rebuild everything for pypy, just what people wanted. We can do the same for pyston.

I'll defer to other core devs for a list of steps to make it happen. I assume building pyston is at the top of the list. We may want to ensure things can be cross-compiled as well.

[jakirkham]

We didn't rebuild everything for pypy, just what people wanted.

Interesting this wasn't my understanding based on the discussion in PR ( conda-forge/conda-forge-pinning-feedstock#225 ) or issue ( regro/cf-scripts#1386 ). For ARM & PPC IIUC this was handled as opt-in as a one-off. Has this changed somewhere for PyPy (or more generally)? If so, where do we maintain the list of things that are opt-in (and how do we opt-out)?

We can do the same for pyston.

This seems reasonable if we have some mechanism for doing that.

I assume building pyston is at the top of the list.

+1

We may want to ensure things can be cross-compiled as well.

To clarify do we care about ARM or PPC here? My temptation would be to skip these on a first pass and stick to x86_64 (unless there is a compelling need for these).

Question for Kevin, et al., are macOS & Windows supported? Or is Pyston Linux only atm?

[beckermr]

Sorry @jakirkham. You are correct that we had a migrator for pypy that tried everything. However large parts of the ecosystem were declared effectively off limits because we did not convert key dependencies.

We can do the same for pyston.

[h-vetinari]

I assume building pyston is at the top of the list.

I don't think we're under huge time pressure here. I'd say that unless there's a recipe by the pyston folks that can be plugged in more or less directly into cf-infra, it would IMO make sense to wait until we can reuse our own LLVM builds, rather than rebuild them as part of pyston (cf. pyston/pyston#180)**, cf. @wolfv's comment above

the 2x llvm builds in there scare me a bit, though :)

Of course this is not a blocker, and all other things being equal, earlier is better, but just wanted to bring that up again.

** while I cannot say how much work this would be on the pyston side, BOLT is already part of LLVM 14, which should reach GA in 2-3 months.

[undingen]

Another Pyston dev here, would be very happy if pyston gets integrated into conda-forge!
Our conda packages are currently build against conda-forge and the recipes can be found here: https://github.com/pyston/pyston/tree/pyston_main/pyston/conda
Or pyston compilation process is unfortunately quite complicated and involves PGO and BOLT which we run on some external python projects.

We did split it very similar to the PyPy packages and use a python_abi package (that's why we also can't use the anaconda channel which has it not set - but I'm sure you already know that). I hope I didn't make any mistakes and this would make official integration easy.

We have our own BOLT package for now but I think the other dependencies all exist in the correct version in conda-forge (llvm+compiler-rt 13.0.1), I tried at first to base our conda repo against the anaconda channel and that caused more problems. I think when LLVM 14 is released we can try to remove our dependency on a specific BOLT version (will try it out soon). But we run in quite a few issues with BOLT previously so we may need to update it in the future to a unreleased version again/integrate some patches.

The released Pyston is currently only supporting Linux on amd64 (no 32 bit), but we are currently working on Arm64 (no 32 bit) support (which landed a few days ago but we did not try building any packages yet) which we will likely release end of next month. Maybe we will also have MacOS support if there is time and we don't run into big problems but we have not tried yet.
For the Arm release we will need to build own packages again but the amd64 we have will be fine because we did not break the ABI. Would be awesome if in the future conda-forge would automatically get build for pyston and not manually like we currently do (which means our packages are not up to date) :).