Proposal: Use Python's limited C API

[drmfinlay]

At the moment, Natlink must be compiled for each minor 32-bit version of Python. It has been a pain-point getting the right PYD file loaded. I wonder if it might be worthwhile to use Python's limited API, documented here and available since Python 3.2. If Natlink could be compiled this way, the same PYD file would work on Python 3.8, 3.9, 3.10 and so on.

It may not be possible to do this with Natlink, I'll need to look into the code first. If it can be done, I would be willing to work on this. I think it would simplify the project and make it easier to maintain going forward.

[drmfinlay]

I've looked into this now. There are a few issues, but I think it's possible.

[drmfinlay]

I've started working on this here: https://github.com/drmfinlay/natlink2/tree/use_py3_stable_abi

My repository (natlink2) doesn't have many of the more recent changes. I used an older revision (without the PyConfig stuff) to reduce the amount of initial work, and to make building a PYD file easier.

Seems to be progressing nicely. The project cannot yet be built without a few prior changes to CMakeLists.txt.

The natConnect() function's optional parameter is a little tricky to deal with using only the limited API. @quintijn, do you know if there is any benefit to passing False (default) to natConnect()? Most users seem to pass True or 1.

[drmfinlay]

I think I am now able to answer my own question. It looks like the only upside to not enabling thread safety is that the Natlink window's "Reload Everything" menu button works.

I have been able to successfully use a Natlink PYD file built for Python 3.4 (stable ABI) with Python 3.8, 3.9 and 3.11. Dragonfly's test suite passes normally. So that's nice. No thread safety yet, however. But then, Natlink doesn't have it on by default anyway.

[LexiconCode]

Presumably, this might make porting to python 11/12 easier.

[drmfinlay]

It should do, yes. Natlink would only need to have one installer for Python 3.

[dougransom]

@quintijn Does unimacro spawn any threads? Or any of the timers?

[quintijn]

That is a too diffucult question, but: 1: I am not aware of that, 2: unimacro occasionally uses the natlinktimer.py feature that I introduced in the recent releases... Let us discuss via zoom at some time, maybe, Quintijn Doug Ransom ***@***.***> schreef op 26 oktober 2023 18:24:32 CEST:

…

@quintijn Does unimacro spawn any threads? Or any of the timers? -- Reply to this email directly or view it on GitHub: #171 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

[drmfinlay]

Sorry, yes, it is a difficult question. I've solved the problem I was having with thread safety. Threads, timers and the natConnect() function are working normally with my changes.

[drmfinlay]

I've got it mostly working now in my natlink2 repo.
https://github.com/drmfinlay/natlink2/tree/use_py3_stable_abi

A PYD can be built and will work out of process on Python 3.3+. As for normal use as a DNS compatibility module, there is a dependency issue with python3.dll. I think this was probably solved a while ago in this repo.

[drmfinlay]

I had to stop working on this in November. The main roadblock here is my limited understanding of how to change the installer logic. Perhaps someone might be able to help me with that at some point?

I wanted to mention, for consideration, an idea I had. I was thinking that, since the installer would work with essentially any 32-bit Python version 3.x, there should be an option in the installer for which version to install/use. Only one version can be used at a time, for complex reasons. Such functionality would also need to exist in the NatLink configurations utilities -- something similar to the "register" button in the old 4.x series GUI.

Let me know if you guys think this is still worth pursuing. I still think the changes would be useful, especially from a maintenance perspective.

[quintijn]

Thanks for all these things, Dane. It is hard for me to judge the things. Also because I am still (and for a long time by now) busy getting a python3 version of Natlink working with all the things around it (like other packager, just now busy getting Unimacro things checked)

Do I understand well, that you make a special python install, just for Natlink to run on? I have other programs having such a dedicated python install. Then the Natlink python install does not have any connection with possible other python installs?

[quintijn]

Dane, do you work with Dragon 16? Can you test if playEvents (and playString) work with you C Limited API??

[drmfinlay]

Sorry for the late reply.

Yes, I normally use Natlink with a system Python installation that isn't used for much else. But that is mainly because Natlink requires 32-bit Python.

I don't work with Dragon 16. However, I can tell you that the issue with playEvents and playString would remain if Python's C limited API were to be used.

Using that API shouldn't do anything other than change the Python DLL requirement of the _natlink_core.pyd file from python310.dll to python3.dll, assuming Python 3.10 is used. That will allow using the same _natlink_core.pyd file with Python 3.8, Python 3.9, Python 3.10, and so on without recompiling.

[drmfinlay]

Thinking about my proposal here, I think it is something that should have been done years ago when the C++ code was made compatible with Python 3. I don't think there's much point now.

[quintijn]

Dane, I am rereading issues of natlink and related. Pffff, what a lot of work you did here, with no concrete result. Sorry I cannot judge all implications of this, but my great appreciations for this work, which will probably not be used then.

In general, we have two types of users,

1. no programmers, they don't care which python is installed, even don't know what python is.
2. programmers, who will use a 64bit version of python. Installing some 32bit version for natlink will not hurt, but more important, "we" can choose which one is suitable, as it will not be used for other applications.

I don't know what I want to say with this, only that some dedicated install will be sufficient.

Greetings, Quintijn

[drmfinlay]

Hi Quintijn, Thank you for your comment. What you say on the two types of Natlink users is true and sounds reasonable to me. A dedicated install is good enough, yes. My work on this issue would have been more valuable when the C++ codebase was being ported to Python 3. In fact, I recall that one of the programmers involved in that mentioned Python's limited API in a comment somewhere. I suppose he found it too tricky to work with. In any case, the work was quite valuable to me as a learning experience.

…

On Thu, 06 Jun 2024 08:53:38 -0700 Quintijn Hoogenboom ***@***.***> wrote: Dane, I am rereading issues of natlink and related. Pffff, what a lot of work you did here, with no concrete result. Sorry I cannot judge all implications of this, but my great appreciations for this work, which will probably not be used then. In general, we have two types of users, 1. no programmers, they don't care which python is installed, even don't know what python is. 2. programmers, who will use a 64bit version of python. Installing some 32bit version for natlink will not hurt, but more important, "we" can choose which one is suitable, as it will not be used for other applications. I don't know what I want to say with this, only that some dedicated install will be sufficient. Greetings, Quintijn -- Reply to this email directly or view it on GitHub: #171 (comment) You are receiving this because you modified the open/close state. Message ID: ***@***.***>

[LexiconCode]

Will this change require Python to be on path?

Anyone with Python developments going to have a 64-bit install on path.

[drmfinlay]

No more than it has to be currently. Python 3's main two DLLs and Natlink's registered server DLL merely need to reside in the same folder. The system will then consider these DLLs over any other matching ones on the path. To me, the simplest solution would be to have the installer copy/hardlink the server DLL from site-packages into the root Python 3 installation folder and register it there. On uninstall it can be unregistered and deleted.

…

On Fri, 27 Sep 2024 11:52:14 -0700 LexiconCode ***@***.***> wrote: Will this change require Python to be on path? -- Reply to this email directly or view it on GitHub: #171 (comment) You are receiving this because you modified the open/close state. Message ID: ***@***.***>