Limited support of the PyPy

[olecsys]

Hello,

I want to suggest limited support of the pywin32 in the PyPy implementation of the Python. Unfortunately PyPy does not provide some functions that pywin32 uses:

• Py_SetProgramName
• Py_Initialize
• PySys_SetArgv

Many Python packages use pywin32 as dependency. I use twisted and it depends on your project. But it does not use win32service, COM objects, etc. So my suggestion is to build all projects that does not require these functions via setup.py. I think platform.python_implementation() is good point to separate CPython and PyPy. I'll be able to make Pull Request, if you'll agree with the approach. And probably I'll need your help in some cases.

[mhammond]

I'm not sure what you are proposing exactly, either in terms of distribution artifacts nor what a patch might look like, so if you could elaborate I could give a thoughtful response.

[olecsys]

Hello @mhammond,
I'll try to explain what I mean. I could execute my twisted application in PyPy 2.7 under Windows OS. I've made some changes to pywin32 setup.py and compiled pywin32 with Python 2.7 VS Build Tools, but it looks like a trick and hack. I just comment out modules, that did not compile. I suggest to add PyPy support in setup.py with more elegant method. I'am sending you my current setup.py.
So I could try to make more elegant code and send you it later.
setup.py.zip

[mhammond]

Why don't the modules compile?

[olecsys]

Main problem is that PyPy CPython compatibility C header Python.h(and PyPy library) does not provide these functions:

• Py_SetProgramName
• Py_Initialize
• PySys_SetArgv

It seems that PyPy maintainers do not plan to implement them for now.

[mhammond]

This sounds fine in theory, but it will depend on what the final patch looks like. It sounds like you really only want the "win32" modules, so maybe you could experiment with conditionally setting ext_modules to only win32_extensions (and we could consider splitting win32_extensions into 2 categories, so things like service support are easier to exclude.

[olecsys]

Yes, I'll try to experiment with ext_modules. Thank you for advice. I'll let you know about results.

[mattip]

Did this experiment get any further? xref https://foss.heptapod.net/pypy/pypy/-/issues/3286, where @SleepyRoy asked for support for the missing c-api functions from PyPy.

[olecsys]

Hello @mattip, actually I did not try to make the experiment then, i.e. I was very busy. But now I could try to make it with pypy3 and we'll see results.

[rchateauneu]

Indeed, just a subset of pywin32 would help a lot. pywin32 is a fantastic tool.

Many thanks.

[stonebig]

trying to buid a PyPy distro, I'm blocked on the Jupyter eco-system part per the missing pywin32... is there any hope ?

[mattip]

I'm blocked on the Jupyter eco-system part

You shouldn't be blocked on the latest versions of the various jupyter libraries. Please file an issue on the repo of the library that is failing to build and ping me.

[stonebig]

apparently, jupyter-core still wants pywin32 : https://github.com/jupyter/jupyter_core/search?q=pywin32&type=code

[mattip]

This is really off-topic for pywin32. Let's discuss it at jupyter/jupyter_core#226

[mattip]

I got CI to pass on the conda-forge feedstock by removing these modules (some small fixes were required to PyPy, they will be part of the upcoming release). Are any of these widely used? How could pywin32 make it clear that these have been removed for PyPy?

• pythoncom, and its dependents adsi, axcontrol, axscript, axdebug, internet, mapi, exchange, shell, propsys, taskscheduler, bits, ifilter, directsound, authorization
• win32ui, and its dependents win32uiole, dde
• servicemanager, pythonservice, and PyISAPI_loader

[mhammond]

Are any of these widely used?

Yes, very widely used.

How could pywin32 make it clear that these have been removed for PyPy?

I don't understand what you mean - I will never remove them from PyPy. Shipping anything with pywin32 in the name but without these modules would be useless to many many pywin32 users, and I'd ask that you do not do that, because I'll be the one handling the bugs about why fundamental things do not work.

[mattip]

OK makes sense. I will not pursue this path.

[mhammond]

Why do you want to ship any of this in the first place? If there's only 1 or 2 things from win32api or similar, then maybe we can work out how to do that, and do it in a way that it would be compatible with the user doing a full install of pywin32?

IOW, I don't object at all to you shipping bits of pywin32, just not a kind of "frankenstein" version :)

[stonebig]

Problem is pywin32 is claimed per jupyter-core , unless they accept cgohlke patch, but apparently they did incorporate it 17hours ago

[isuruf]

If there's only 1 or 2 things from win32api or similar, then maybe we can work out how to do that, and do it in a way that it would be compatible with the user doing a full install of pywin32?

@mhammond, do you have any ideas on how to do this? If so, I can try and send a PR.

[mhammond]

@mhammond, do you have any ideas on how to do this? If so, I can try and send a PR.

No - I've no idea what parts people want to ship and I've no idea how jupyter's build process works. There's no PR possible for pywin32 that would address this.

[isuruf]

@mhammond, I'd like to ship all the parts that can be built using PyPy and only disable modules that use features that PyPy doesn't support. jupyter now doesn't need pywin32, but there are other packages that do need pywin32.

[isuruf]

FYI, I only need win32api, win32con, win32event, win32process, win32com.shell for a project.

[mhammond]

fair enough - but sorry, I've no idea how pypy works nor how you would do that. The initial comment here talks about pypy missing some really fundamental APIs and I'd be reluctant to take a complicated PR that works around these limitations - so a better approach might be to fix pypy?

[isuruf]

PyPy is an alternative python implementation. When building a python extension, the extension has suffix pypy39-pp73-win_amd64.pyd instead of cp39-win_amd64.pyd for CPython (reference implementation). So there's no possibility of a pywin32 built for PyPy clashing with a pywin32 built for CPython.

A PR would simply disable building the modules that require features that PyPy doesn't support. For eg: https://raw.githubusercontent.com/mattip/pywin32-feedstock/182ec7282aa78027bfcae08e0667c64ff4ad7587/recipe/patches/0001-remove-modules-needing-pythonservice.cpp-for-pypy.patch

I asked @mattip who is a core contributor of PyPy recently what the status of PyPy implementing these missing features at
https://foss.heptapod.net/pypy/pypy/-/issues/3286#note_162769. I'll try to go that route and maybe we'll see support for these features in a new PyPy release.

[mhammond]

I'd be fine with a relatively simple PR touching only setup.py

[isuruf]

Thanks.

@mattip, since it's your patch, would you like to send the PR?

[mattip]

I have started working on this in a branch. The changes to avoid the com_extensions and the pythonwin_extensions is straightforward. I also added PyPy to the CI matrix, and some tests will require skipping. It seems PyPy is still missing some basic functionality, like PyUnicode_EncodeCodePage, and maybe more.

[mhammond]

like PyUnicode_EncodeCodePage,

If only some of those obscure strings functions are causing problems, many of them can probably be killed now that python 2 support has been dropped. They might not all be trivial to kill, but it is possible. I'll try and clean some of these up soon, but it's likely to take me a few weeks to get to this.

[x151973]

Any progress here? I have same issue with pypy that copying bmp raw data to windows clipboard, using win32clipboard which is a part of pywin32

[Avasam]

@x151973 #1860 removed PyUnicode_EncodeCodePage. For everything else, go follow those issues:

• Can not compile win32api with pypy pypy/pypy#1720
• Missing Py_SetProgramName, Py_Initialize, PySys_SetArgv* pypy/pypy#3286
• Implement Py_Initialize pypy/pypy#3836

[mattip]

I updated the branch, see mattip#2, but without Py_Initialize I fear I cannot make much headway. Implementing Py_Initialize is a big project, see pypy/pypy#3836 and unlikely to move forward without some help either in terms of sponsorship or coding (I can mentor someone who would like to do the work).