Use pdm as a helper and a build backend #140

getzze · 2024-10-29T15:42:47Z

fixes #127, #128
Supersedes #138

Use pyproject.toml with pdm backend.
Use src (avoid some errors while running tests) and scripts folders.
Bump minimal python version to 3.9 (although it should work fine on py3.8, but it just reached EOL). Support python 3.13.
Build wheel with bundled libmediainfo for MacOS arm64 and for Linux (x886_64 and arm64).
Fix linting and type errors in source and tests.

pdm (like most "new" python build backend like hatch, poetry, flit) do not support C-extension yet. But for this package, building the wheel for different platforms only means downloading the correct flavor of the library. Hence a universal wheel is built and is later tagged with the correct platform tag using wheel.
However it's very simple to create scripts with pdm, which can be used to simplify the download process (future PR).

I don't have an appveyor account so I cannot run the tests.

Future PRs:

use pdm scripts to download the libraries.
switch from black-isort-flake8-pylint to ruff
use pre-commit hook.

sbraz

Wow thanks a lot, that is a great PR! pbr does seem to make wheel generation much easier!
I think I've asked basically all the questions I needed.
Once I'm OK with everything, do you mind if I reword some commits to add more details (e.g. why support for older Pythons was dropped, why method order was changed, etc.)?

appveyor.yml

sbraz · 2024-10-29T20:53:13Z

appveyor.yml

+      pdm build --no-sdist
+      wheel tags --remove --platform-tag=manylinux_2_34-x86_64 dist/*-py3-none-any.whl
+      # wheel arm64
+      mv -f arm64/lib/libmediainfo.so.0.0.0 pymediainfo/libmediainfo.so.0


If upstream changes the version, the name of the actual file but .so.0 will still be a symlink, so this would be more future-proof?

Suggested change

mv -f arm64/lib/libmediainfo.so.0.0.0 pymediainfo/libmediainfo.so.0

cp arm64/lib/libmediainfo.so.0 pymediainfo/libmediainfo.so.0

I don't understand what you mean, the .so.0.0.0 is the real file, the .so.0 is just a symlink to it. Isn't it better to copy the real file?

In the code, the default is to use library_paths = ("libmediainfo.so.0",). This means that this library (pymediainfo) expects it to exist. If it doesn't, there's something wrong and I'll have to update the code, not only the CI.
However, nothing guarantees that the real file is named libmediainfo.so.0.0.0. That's why I thought it would be better to copy libmediainfo.so.0 (cp resolves the link to copy the actual file).
Please let me know if this is clear.

OK, but it's renamed to .so.0 (mv moves and renames files). It's a common practice to have the real lib file with full version and symlink to .so.0 and .so. Therefore I doubt mediainfo will change the naming scheme, but as you prefer.

Oh I know your command renames the file to what is expected.
All I'm saying is that there is no guarantee that upstream won't change the filename later from libmediainfo.so.0.0.0 to something. On the other hand, since libmediainfo.so.0 is the SONAME, I think it is less likely for them to rename libmediainfo.so.0 without a major reason.

objdump -p libmediainfo.so.0 | grep SONAME SONAME libmediainfo.so.0

@JeromeMartinez please let me know if my idea of copying the .so.0.0 by resolving the symlink from .so.0 makes more sense than directly copying the .so.0.0. Also, do you have an opinion on us bundling the AWS Lambda library into the Python wheels?

src/pymediainfo/__init__.py

tox.ini

pyproject.toml

src/pymediainfo/__init__.py

use exact filename Co-authored-by: Louis Sautier <[email protected]>

getzze · 2024-10-30T00:25:10Z

Please modify this PR and its commits as much as you want!

Tohrusky · 2024-10-30T00:29:25Z

Considering migrating to github actions? I think it's more simple for CI/CD

sbraz · 2024-10-30T00:37:59Z

Considering migrating to github actions? I think it's more simple for CI/CD

I considered it at some point but I like the idea of having CI/CD independent from the code. If I ever want to move this project out of GitHub, I can still keep using AppVeyor.
Also making the CI work took quite some time, then Travis basically stopped being free so I had to migrate to AppVeyor… I'm not sure I want to migrate a second time unless it's really helpful.
Let's first make these wheels work. Then, we can decide what we want to do with the CI/CD.

sbraz · 2024-11-17T13:29:15Z

@getzze I'm reworking your commits right now. I'm removing ruff/coverage/typos from setup.py (basically, everything not used by the CI/CD right now). I'll do it in my own branch so it won't get lost for future PRs.

Build wheel with bundled libmediainfo for MacOS arm64 and for Linux (x886_64 and arm64). Fixes: #127 Fixes: #128 Fixes: #140

setup.cfg

Build wheel with bundled libmediainfo for MacOS arm64 and for Linux (x86_64 and arm64). Instead of installing MediaInfo via apt to run tests, use the AWS Lambda library file that we bundle into Linux wheels, this simplifies the CI code. Fixes: #127 Fixes: #128 Fixes: #140

sbraz · 2024-11-17T14:34:11Z

pyproject.toml

@@ -0,0 +1,199 @@
+# https://packaging.python.org/en/latest/specifications/pyproject-toml/
+[build-system]
+requires = ["pdm-backend", "wheel>=0.42"]


Should wheel really be here? wheel is not required for the sdist, right? I'm afraid this is going to cause issues to Linux distributions that package pymediainfo.

It's needed to build the wheels but not the sdist. Why do you think it will be an issue to package it with Linux distribution? python-wheel is part of the new build workflow, cut off from setuptools, I am sure it's packaged for all the distros.

We at Gentoo don't like declaring dependencies when they're not actually needed. Isn't there a better way to declare it as a dependency only for the wheels? Ping @mgorny.

Ok, fair enough.
The dependencies defined in build-system are used when you do pip install .
In this case you don't need to build a wheel with the libs bundled. So you can remove wheel from these deps.
What can be done then, is to put the wheel dep in a pdm script that builds the bundled wheels (I'm preparing this PR).

getzze · 2025-02-07T11:29:57Z

@sbraz Hey, any update for this PR? Let me know if you need help to finish it.

sbraz · 2025-02-07T15:45:32Z

Hi @getzze, I will try to finish this over the weekend. I'm very sorry for how long it's taking.

Build wheel with bundled libmediainfo for MacOS arm64 and for Linux (x86_64 and arm64). Instead of installing MediaInfo via apt to run tests, use the AWS Lambda library file that we bundle into Linux wheels, this simplifies the CI code. Fixes: #127 Fixes: #128 Fixes: #140

getzze added 7 commits October 29, 2024 11:41

update python versions

07a38f3

use src

9356838

use pyproject

32b3e8a

correct linter and type errors

c848f25

fix types

f3f9ea1

fix linter and tests

7273021

fix tests

de7fd55

sbraz reviewed Oct 29, 2024

View reviewed changes

Update appveyor.yml

4f07f74

use exact filename Co-authored-by: Louis Sautier <[email protected]>

sbraz mentioned this pull request Oct 30, 2024

Publish wheel for macos-arm64, manylinux-x86_64 and manylinux-arm64 #138

Closed

sbraz pushed a commit that referenced this pull request Nov 17, 2024

Switch to pyproject + pdm, add Python 3.13, drop 3.7 and 3.8

813aff2

Build wheel with bundled libmediainfo for MacOS arm64 and for Linux (x886_64 and arm64). Fixes: #127 Fixes: #128 Fixes: #140

sbraz reviewed Nov 17, 2024

View reviewed changes

setup.cfg Show resolved Hide resolved

sbraz reviewed Nov 17, 2024

View reviewed changes

sbraz closed this in bf780d1 Feb 11, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use pdm as a helper and a build backend #140

Use pdm as a helper and a build backend #140

getzze commented Oct 29, 2024 •

edited

Loading

sbraz left a comment

sbraz Oct 29, 2024

getzze Oct 29, 2024

sbraz Oct 29, 2024 •

edited

Loading

getzze Oct 30, 2024

sbraz Oct 30, 2024

getzze commented Oct 30, 2024

Tohrusky commented Oct 30, 2024

sbraz commented Oct 30, 2024

sbraz commented Nov 17, 2024

sbraz Nov 17, 2024

getzze Nov 17, 2024

sbraz Nov 17, 2024

getzze Nov 17, 2024

getzze Nov 19, 2024

getzze commented Feb 7, 2025

sbraz commented Feb 7, 2025

	mv -f arm64/lib/libmediainfo.so.0.0.0 pymediainfo/libmediainfo.so.0
	cp arm64/lib/libmediainfo.so.0 pymediainfo/libmediainfo.so.0

Use pdm as a helper and a build backend #140

Use pdm as a helper and a build backend #140

Conversation

getzze commented Oct 29, 2024 • edited Loading

sbraz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbraz Oct 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

getzze commented Oct 30, 2024

Tohrusky commented Oct 30, 2024

sbraz commented Oct 30, 2024

sbraz commented Nov 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

getzze commented Feb 7, 2025

sbraz commented Feb 7, 2025

getzze commented Oct 29, 2024 •

edited

Loading

sbraz Oct 29, 2024 •

edited

Loading