multicorn2 in pypi.org

[luss]

I suggest we add this to our roadmap of ToDo items. I think we should call it multicorn2 since we don't own the 'multicorn' project.

[ShaheedHaque]

This would be fantastic! Just a reminder though of the challenges involved compared to a normal Python-with-binary-bits, in the worst case of an unprivileged venv user:

1. A compiled library, possibly for one of the multiple installed PG versions, installed where the PG server code expects it, i. e. using privs.
2. A second compiled library like the first, containing the utils.py support (e. g. log_to_postgres) where the multicorn server wrapper is installed, i. e. using privs.
3. The same as 2. but installed in there user's venv. Why is this needed? Because the Python FDW code in server use almost certainly shares definitions with the client use, and modern pip/venv cannot use 2.

For 1. I have seen Java-based packages which select the Java instance using an env var. That's ugly, but it work. I guess the privs will have to use 'sudo pip' and --break-system-packages for 2.

[luss]

Good points. I've never personally done pypi for something with c code. In the short run, any pg distro's that want multicorn will need to package it such that it is installable in their distro. I think we could likely get pgdg to build us once we get reliable & cleanly compiling and working on all current pg versions. We are on our way... Viva Multicorn!! :-)

…

On Mon, May 27, 2024 at 1:29 AM ShaheedHaque ***@***.***> wrote: This would be fantastic! Just a reminder though of the challenges involved compared to a normal Python-with-binary-bits, in the worst case of an unprivileged venv user: 1. A compiled library, possibly for one of the multiple installed PG versions, installed where the PG server code expects it, i. e. using privs. 2. A second compiled library like the first, containing the utils.py support (e. g. log_to_postgres) where the multicorn server wrapper is installed, i. e. using privs. 3. The same as 2. but installed in there user's venv. Why is this needed? Because the Python FDW code in server use almost certainly shares definitions with the client use, and modern pip/venv cannot use 2. — Reply to this email directly, view it on GitHub <#56 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMWOHURCTD6IOO3NILMG3TZELADVAVCNFSM6AAAAABIFEVLEKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TKNRWGI3TS> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[whummer]

Having the library published on pypi would be awesome! (I also just raised this in #61 before). Just to confirm, when you mention binaries, you're referring to the multicorn.so shared library that gets built when running make install, is that right?

I'm trying to understand the need to package this binary file into the pypi package, maybe it wouldn't even be required in a first iteration? 🤔

The way we've currently set up our project using multicorn is to provide the precompiled binaries (i.e., multicorn.so) for the Postgres extension (for a particular Postgres version, and only for Linux AMD64/ARM64), and then install the Python code from source. The last step would be quite a bit easier if there was a pip package available (to allow us to create our custom data wrapper which derives from multicorn.ForeignDataWrapper).

I think providing pre-compiled binaries for all architectures / operating systems / Postgres versions may be a great longer term goal, but sounds like quite a large undertaking. Maybe a pypi package with only the Python sources could be a great starting point, or do you see major limitations with that @ShaheedHaque @luss ?

[ShaheedHaque]

Having the library published on pypi would be awesome! (I also just raised this in #61 before). Just to confirm, when you mention binaries, you're referring to the multicorn.so shared library that gets built when running make install, is that right?

In #56 (comment), bullet 1 refers to multicorn.so, and bullet 2 refers to utils.so.

I'm trying to understand the need to package this binary file into the pypi package, maybe it wouldn't even be required in a first iteration? 🤔

The Python bits are useless without the multicorn.so, at a minimum, since PG invokes it to invoke your Python FDW code. Given the need to solve multicorn.so, one may as well do utils.so since without it, diagnostics stop working.

The way we've currently set up our project using multicorn is to provide the precompiled binaries (i.e., multicorn.so) for the Postgres extension (for a particular Postgres version, and only for Linux AMD64/ARM64), and then install the Python code from source. The last step would be quite a bit easier if there was a pip package available (to allow us to create our custom data wrapper which derives from multicorn.ForeignDataWrapper).

Guess what...that's exactly what I need too!

I think providing pre-compiled binaries for all architectures / operating systems / Postgres versions may be a great longer term goal, but sounds like quite a large undertaking. Maybe a pypi package with only the Python sources could be a great starting point, or do you see major limitations with that @ShaheedHaque @luss ?

I'm not even thinking of pre-compiled binaries. Doing the compile step from PIP (implicitly requiring the correct Postgres headers, and needed build tools to be installed) would be the goal. Technically, it is probably not all that hard to do the steps I outlined, but it requires:

• Turning the current build system "inside out", i.e. where PIP is in change rather than make.
• A clear model for how/where sudo-versus-user does the multiple installs.
• A way, as I noted, for the user to select the "right" PG to build against.

Probably not all that hard, but I don't currently have time to spare for this.

[luss]

hmmm. maybe PyPi shouldn't be our target. Extensions that use multicorn2 should be pure python and therefore are ideally installed w PyPi. We can work to get Multicorn2 built by the PGDG group so that our extension can easily be installed from standard community Postgres installations (and from pgEdge sandboxes).

[ShaheedHaque]

hmmm. maybe PyPi shouldn't be our target. Extensions that use multicorn2 should be pure python and therefore are ideally installed w PyPi. We can work to get Multicorn2 built by the PGDG group so that our extension can easily be installed from standard community Postgres installations (and from pgEdge sandboxes).

I'm not familiar with these possibilities. My end requirement is a scripted install (hopefully simpler than the current script we launch VMs with).

[luss]

I still like the PIP INSTALL idea. I just personally have never packaged something for PIP but if anyone has... join the discussion and submit a patch for 3.1