Cython Support

[htnjack]

Is your feature request related to a problem? Please describe.
I've been trying to integrate a project that depends on some cython extensions to a monorepo. It was pretty hard and time consuming for me as I don't know much about Cython and I couldn't find minimal examples online.

After trying many approaches (including plugin development) I found that pants and the PEP-517 support allows to build Wheel distributions "natively" using a custom setup.py to compile .pyx files.

Describe the solution you'd like
I'd like to have full cython support integrated with pants, or at least have a plugin for cython support. Ideally, both .pyx compilation and Pure Python mode should be supported.

Describe alternatives you've considered
I considered developing a plugin for my specific use-case, however it's a daunting task for me and it would be very time consuming.
I know some people have done so, and @sureshjoshi has interest in prototyping it (sureshjoshi/pants-plugins#14). I'm not sure how popular Cython is right now and how much interest there is for this feature, but I'm willing to spend some time looking into this.

Additional context
I whipped up a minimal example that should showcase Cython support. At the moment it can build a wheel using a python_distribution target.
You can find it at https://github.com/houtenjack/pants-cython-minimal-example

[sureshjoshi]

Thanks for the example!

Not sure if you're on the Slack channel (edit: just saw you posted this there first), but there is a more lively discussion about these kinds of things there, with some idea swapping. I've seen cython mentioned a few times (including by me - I have a very hardcoded-type plugin for an internal project, but it works for my simple use cases).

I haven't thought this through too much on how that plugin could be made "correct", but I think some internal Pants functionality would need to be exposed.

Currently, Pants allows setting custom setup.py kwargs, but for stuff like cython and mypyc, I think we need to allow generation of the whole setup.py. Perhaps there is a way using the existing hooks, but if there is, I couldn't figure it out.

To give an idea of what that looks like for mypyc, I have an example here
https://github.com/sureshjoshi/pants-plugins/blob/472051c4d014fbea579664016578f92fe1f36358/pants-plugins/experimental/mypyc/rules.py#L120-L141

This plugin requires a custom Pants fork with this added functionality: 9f9cbe0

I think if custom setup.py files could be generated dynamically, then basic cython, mypyc, and maybe nuitka plugins could be supported somewhat trivially (not all the edge cases though, naturally).

[htnjack]

I've seen your mypyc plugin and that's what I used as reference for trying my own cython one, thank you for making that repo (and this awesome blog post)!

For my use case, I am fine with writing my own setup.py files, but I agree that supporting setup.py generation would be a good idea for a generic implementation.

Something that @benjyw mentioned is that it would be fairly easy and extremely helpful to have a cython_sources target that collects all the cython specific files (instead of including them as resources).

I'd be happy to try setting up a plugin for this issue, what would be a good reference? Should this be in this repo, or in a dedicated one?

Just note that I'd be focusing on my use cases, so I'm hoping to get enough interest and contributions to build a proper generic plugin.

[sureshjoshi]

Hey @houtenjack, no problem! Happy to help!

For creating a custom plugin, the two main options are forking Pants and then just working in that repo. I find that a bit overwhelming personally, because there is so much stuff when I'm jumping around files.

The other option is an in-source plugin (which is what my pants-plugins repo does). This is a bit tighter to navigate through, and easier to make example apps - however, if you need to make modifications to the Pants source, then you'll need to use pants_from_sources functionality.

My workflow is to write in-repo plugins, and then if there is a PR - perform that after everything works.