Enhance shared library support for AIX

[KamathForAIX]

In AIX, when we build packages like SciPy or matplotlib using pip3, currently we get a .so file not found error. This is because the .so file is in the archive and we archive shared libraries in AIX.

For example we receive an error like below when we use pip3 install matplotlib:-
Traceback (most recent call last):
File "/opt/freeware/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in
main()
File "/opt/freeware/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/opt/freeware/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 160, in prepare_metadata_for_build_wheel
whl_basename = backend.build_wheel(metadata_directory, config_settings)
File "/tmp/pip-build-env-nyo76fqj/overlay/lib/python3.9/site-packages/mesonpy/init.py", line 1008, in wrapper
return func(*args, **kwargs)
File "/tmp/pip-build-env-nyo76fqj/overlay/lib/python3.9/site-packages/mesonpy/init.py", line 1073, in build_wheel
return project.wheel(out).name
File "/tmp/pip-build-env-nyo76fqj/overlay/lib/python3.9/site-packages/mesonpy/init.py", line 928, in wheel
file = self._wheel_builder.build(directory)
File "/tmp/pip-build-env-nyo76fqj/overlay/lib/python3.9/site-packages/mesonpy/init.py", line 497, in build
self._install_path(whl, src, dst)
File "/tmp/pip-build-env-nyo76fqj/overlay/lib/python3.9/site-packages/mesonpy/init.py", line 453, in _install_path
wheel_file.write(origin, destination.as_posix())
File "/tmp/pip-build-env-nyo76fqj/overlay/lib/python3.9/site-packages/mesonpy/_wheelfile.py", line 94, in write
with open(filename, 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-install-q1huanuy/contourpy_da5341e8ada44e41863d6e5dd4b423f8/.mesonpy-xt0p8wzf/src/_contourpy.cpython-39.so'
[end of output]

While using python3 in AIX, the python modules should have the .so files instead of a .a file just like other platforms. Other normal packages like glib2 can have archive having the shared library.

So we need to enhance meson to support this behaviour in AIX where we can archive shared library or have only a shared library object and give control to AIX users to choose between them.

In-order to support the same in AIX, we propose to export an environment variable AIX_SO_ARCHIVE such that if environment variable is set we will need to archive the shared library and if not set we need not archive.

So any package that needs an archived shared library in AIX will need to set the AIX_SO_ARCHIVE to 1, otherwise the default behaviour should be not to archive shared libraries.

[eli-schwartz]

and give control to AIX users to choose between them.

In-order to support the same in AIX, we propose to export an environment variable AIX_SO_ARCHIVE such that if environment variable is set we will need to archive the shared library and if not set we need not archive.

This sounds like something that is specific to the project, not to the user. Am I missing something?

Specifically, AIX presumes the use of archives in general, but python is a special case because it uses a dlopen() mechanism and wants .so files?

[KamathForAIX]

and give control to AIX users to choose between them.
In-order to support the same in AIX, we propose to export an environment variable AIX_SO_ARCHIVE such that if environment variable is set we will need to archive the shared library and if not set we need not archive.

This sounds like something that is specific to the project, not to the user. Am I missing something?

Specifically, AIX presumes the use of archives in general, but python is a special case because it uses a dlopen() mechanism and wants .so files?

Yes. The python3 projects need .so files. And in general we use archives.

[eli-schwartz]

So it seems to me that:

1. We should not permit the user to choose
2. The python module should automatically set an attribute on the build.SharedModule object instance that makes it use a plain .so.

Expecting end users who are building a project to manually set environment variables to make it work is a bad user experience -- furthermore, it means that you can't build a shared C library and also a python module, in the same project -- because the environment variable has no granularity and ends up applying to both.

There's another general problem with an environment variable -- we can't guarantee that it will be set both at setup time and at install time, so we'd need to read it from the environment and write it into the coredata / install data on the first setup. Or better would be to make it a machine file property. But that would only be relevant if we wanted users to be able to control it outside of the project build definition.

[KamathForAIX]

Hi @eli-schwartz

So it seems to me that:

1. We should not permit the user to choose
2. The python module should automatically set an attribute on the build.SharedModule object instance that makes it use a plain .so.

Yes your points are right. I agree with you. We did start thinking about the second point initially.

What you are telling is we can have a variable in SharedModule or SharedLibrary class and we can read it via user input in the shared_library function of meson.build file of projects using kwargs. In this project we can use this known_shlib_kwargs or known_shmod_kwargs for the same. Kindly correct me if I am are wrong.

The main point with the second approach is that we also have to patch Open source python packages meson.build files to make it work in AIX. As more projects are adopting meson build system, this might be a tedious work. That is the main reason we went with the environment variable approach instead.

Expecting end users who are building a project to manually set environment variables to make it work is a bad user experience -- furthermore, it means that you can't build a shared C library and also a python module, in the same project -- because the environment variable has no granularity and ends up applying to both.

There's another general problem with an environment variable -- we can't guarantee that it will be set both at setup time and at install time, so we'd need to read it from the environment and write it into the coredata / install data on the first setup. Or better would be to make it a machine file property. But that would only be relevant if we wanted users to be able to control it outside of the project build definition.

I agree with all your points here. Thank you for suggesting this. I did not think deeply about a case where we will have both C and python module. Appreciate your analysis. So I feel environment variable may not be the best solution after this.

So how do you think we should solve this. Should we go with the second point you mentioned and use it the way I just discussed above if I understood it correctly.

Kindly guide us with your experience so that we can have the best solution for this problem.

Have a nice day ahead.

[eli-schwartz]

What you are telling is we can have a variable in SharedModule or SharedLibrary class

Yes.

and we can read it via user input in the shared_library function of meson.build file of projects using kwargs. In this project we can use this known_shlib_kwargs or known_shmod_kwargs for the same. Kindly correct me if I am are wrong.

The main point with the second approach is that we also have to patch Open source python packages meson.build files to make it work in AIX. As more projects are adopting meson build system, this might be a tedious work. That is the main reason we went with the environment variable approach instead.

No. The py.extension_module() function doesn't need a kwarg, because meson can just automatically set it whenever extension_module is used.

The only time that you'd need to patch open source projects' meson.build files to make it work in AIX is if there is software other than python extensions that needs this to be done. Is there?

[KamathForAIX]

Hi @eli-schwartz,

Thank you for your response. I appreciate your guidance. I have a few things which I am still unclear about what you are suggesting that I want to clear so we are thinking the same way to solve this.

No. The py.extension_module() function doesn't need a kwarg, because meson can just automatically set it whenever extension_module is used.

What you are saying is python3 modules will use an extension module which will then call the extension module internally in meson defined under modules/python3.py. So I see a known_shmod_kwargs imported here and we can easily use this to set the variable to not archive in the sharedLibrary class. Am I correct here?

So ideally I expect python extension libraries to have something like this in the meson.build files.

py.extension_module('tachyon',
  '../tachyon_module.c',
  c_args: '-DMESON_MODULENAME="tachyon"',
)

But in packages like SciPY {https://github.com/scipy/scipy} I did not find them use py.extension_module().

I only see

py_mod = import('python')
py3 = py_mod.find_installation(pure: false)
py3_dep = py3.dependency()

In fact when I tried to debug if it did use and did not succeed. Kindly guide me with your experience and ideas on if I missed understanding something here or did my way to debug what you were trying to tell me in the codebase went wrong.

The only time that you'd need to patch open source projects' meson.build files to make it work in AIX is if there is software other than python extensions that needs this to be done. Is there?

There can be a situation where in there can be C libraries which might have plugins for python so that the same library can be used in python as well. These plugins might need a .so file instead of archive. However patching them may not be a huge task. That is the only thing we have in mind. Otherwise there is not be a software other than python extensions so far that we have come across.

Have a nice day ahead.

[KamathForAIX]

Respected @eli-schwartz and community members,

Kindly guide me with respect to my previous reply. Somewhere I feel I did not get you completely and need to learn more about it. Your knowledge, expertise and guidance will help us solve this issue in the best possible way.

Waiting for a reply,
Have a nice day ahead.

[KamathForAIX]

Respected @eli-schwartz and community members,

Hi,

I tried using an attribute in AIX via the build.SharedModule inorder to stop meson from archiving a shared library if the project is a python package. As per your suggestion I went on to set it in the extension module of the python3 modules file in meson. Using meson to build scipy however this did not work. while I tried to debug by setting a breakpoint in the extension module function, it did not hit that place. Possibly it did not use it or I did not understand this correctly on what you meant by py.extension () module.

So far I did not succeed in solving this issue and also, possibly did not get completely on what you were trying to tell me. Kindly give me your further suggestions and correct me if I have made a mistake in understanding so, so that we can solve this problem for meson and AIX.

Have a nice day ahead.

Thanks and regards,

[eli-schwartz]

Apologies for the lateness, this slipped my notice for a bit.

But in packages like SciPY {https://github.com/scipy/scipy} I did not find them use py.extension_module().

The py is a variable assignment and extension_module is a method call on the assigned object. So for SciPy you can look for py3.extension_module instead.

So it should work for SciPy too.

[KamathForAIX]

Respected @eli-schwartz and community members,

Hi,

Thank you for your response.

But in packages like SciPY {https://github.com/scipy/scipy} I did not find them use py.extension_module().

The py is a variable assignment and extension_module is a method call on the assigned object. So for SciPy you can look for py3.extension_module instead.

So it should work for SciPy too.

I have figured this out. SciPy has internally many other meson.build files which uses py3.extension modules. This is calling the extension_module_method () in python.py module. Packages like glib2 which are non python modules do not use them. So this becomes a nice differentiation.

We are okay with this design. This is a really nice idea. Thank you for suggesting this to us. I appreciate this a lot and your guidance. By this AIX folks will not have to use any environment variables making it a smooth user experience.

Requesting your permission to create a pull request. I have the tentative fix for this issue same using a variable in the build.SharedLibrary for which I will need your guidance so that we can solve this for AIX and meson..

Kindly let me know.

Have a nice day ahead.

Thanks and regards.

[eli-schwartz]

I have the tentative fix for this issue same using a variable in the build.SharedLibrary

That seems reasonable to me.

Do you need it for the meson.build DSL function shared_module() too?

[KamathForAIX]

I have the tentative fix for this issue same using a variable in the build.SharedLibrary

That seems reasonable to me.

Do you need it for the meson.build DSL function shared_module() too?

Hi @eli-schwartz

We were attempting to build packages like scipy or conttourpy that were using Shared_library. So those one's are getting built with the fix to only shared_library

We understand that packages using shared_module will also need to be stopped from archiving shared libraries cz finally the dl_open () will look for a .so file there as well. But we do not know of any package that we can test, build and learn on how shared_module works. From your knowledge, kindly suggest one from which we can try to understand and raise another PR perhaps if you agree trying to solve that too.

[eli-schwartz]

A few that come to mind are:

• postgres
• libvips
• irssi
• p11-kit

[KamathForAIX]

Respected @eli-schwartz and community members,

Hi,

Thank you for your reply and patience.

A few that come to mind are:

• postgres
• libvips
• irssi
• p11-kit

So in AIX we use p11-kit. We do not want to archive in case shared_module is getting used. And that ensures it works well.

To conclude we have three things,

• In AIX, package by default archives shared library when shared_library is used. We built glib2 for this.
• A package should not archive shared library when py3.extension_module () is used due to the dl_open () mechanism of python. We built scipy, conttourpy for this.
• If shared_module is used then we should not archive the .so coming from shared_module (). We tried p11-kit for the same.

So to do the same we use a variable in SharedLibrary class which is by default set to True to archive shared library. When py3.extension_module () is used then we set this variable to False. This ensures for python3 modules we do not archive. Apart from this we also check if the build instance target is a shared_module. If it is then do not archive the shared library.

This is the design we are thinking. Let me know what you think.

If you are okay with this design, kindly grant me the permission to create a Pull Request. I have the solution in AIX which we want to contribute for Meson community and AIX with your guidance.

Regards,
Aditya Kamath.

[eli-schwartz]

This looks like a great solution, thanks for investigating the options. I figured p11-kit would be the easiest to check, actually. :D

A PR implementing this would be great.

So to do the same we use a variable in SharedLibrary class which is by default set to True to archive shared library. When py3.extension_module () is used then we set this variable to False. This ensures for python3 modules we do not archive. Apart from this we also check if the build instance target is a shared_module. If it is then do not archive the shared library.

It may also be possible to just set the variable to false when deriving the SharedModule class, but I haven't checked.

[KamathForAIX]

Hi @eli-schwartz ,

Kindly review the PR. Let me know what you think.

This looks like a great solution, thanks for investigating the options. I figured p11-kit would be the easiest to check, actually. :D

A PR implementing this would be great.

Yes, thank you for suggesting this. I appreciate the same. :)

Thanks and regards,
Aditya.