Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PyPI updating strategy #581

Closed
cookpa opened this issue Mar 16, 2024 · 23 comments
Closed

PyPI updating strategy #581

cookpa opened this issue Mar 16, 2024 · 23 comments
Labels
distribution Issues with CI, PyPI, etc question

Comments

@cookpa
Copy link
Member

cookpa commented Mar 16, 2024

As discussed in #578 and #580

From @ncullen93:

looks like it's not the specific build size but just the total storage on pypi being over 10gb? So some old wheels need to be deleted? For what it's worth, I really don't think you need to build wheels for each python distribution specifically. I think it's more common to just build one wheel for "3.x" - see e.g., the pypi workflow I use elsewhere.

I don't have permission to delete old wheels. I am not attached to building every python, I don't know if there's stats on which versions get downloaded the most but will look into it.

What do others (devs and users) think about building fewer wheels?

@cookpa cookpa added question distribution Issues with CI, PyPI, etc labels Mar 16, 2024
@cookpa
Copy link
Member Author

cookpa commented Mar 17, 2024

Via pypistats, over the last 180 days:

image image

@cookpa
Copy link
Member Author

cookpa commented Apr 2, 2024

@stnava with the bug fixes at 0.5.1 it might be good to update PyPI. Python 3.8 will reach EOL in October 2024, so maybe that can be dropped.

Hopefully we will get more space to play with once the more efficient wheels are available via #591 .

@ncullen93
Copy link
Member

Yes the new wheels will be very small. It should be done within a few weeks.

@stnava
Copy link
Member

stnava commented Apr 2, 2024

ok - do you need pypi admin access @cookpa ?

@cookpa
Copy link
Member Author

cookpa commented Apr 2, 2024

I think I have to be an owner to change things, yes. Whatever I am now lets me see the project under my project, but I can only view.

@stnava
Copy link
Member

stnava commented Apr 2, 2024

you got it

@ncullen93
Copy link
Member

@cookpa can we try to build wheels now and see the size? When I build locally using python -m build it's only 16mb but that seems too small.

@cookpa
Copy link
Member Author

cookpa commented May 14, 2024

@cookpa
Copy link
Member Author

cookpa commented May 14, 2024

Windows is failing because it explicitly calls setup.py, let's see if the others build

@cookpa
Copy link
Member Author

cookpa commented May 14, 2024

@ncullen93 I got the Linux wheels to build, the artifacts are named funny but I can fix that. I installed the python 3.10 Linux wheel and it runs sucessfully! Amazing that it's only 17Mb

@cookpa
Copy link
Member Author

cookpa commented May 14, 2024

On Intel Mac, there is a problem with cibuildwheel, it is getting into trouble with the MACOSX_DEPLOYMENT_TARGET. I think it might relate to this known issue with scikit-build

pypa/cibuildwheel#952

Bumping the version of scikit-build and / or cibuildwheel might help

@ncullen93
Copy link
Member

ncullen93 commented May 14, 2024

Yeah I see the small size is also replicated on the ci build. Hoping it is for real. I will try to fix the wheel issues.
edit: but I will wait to see if your PR fixes it

@cookpa
Copy link
Member Author

cookpa commented May 14, 2024

I'm hoping that setting MACOSX_DEPLOYMENT_TARGET=10.14 in the environment will solve the problem. Somewhere it is setting a default of 10.9 and not respecting the pyproject.toml entry.

I may also have to downgrade cibuildwheel to 2.16.2 again, because the latest version appears to break the package delocalization on arm64.

@cookpa
Copy link
Member Author

cookpa commented Aug 6, 2024

Wheels are looking good, they fall over occasionally but mostly seem to build. It's been a long time since a release made it to PyPI, it would be good to get a new one up.

To do:

  • Make space on PyPi. They discourage deleting releases, but it seems like we've done it before, and probably have no choice but to delete at least 0.3.8. The new wheels are so much smaller, we can get 10-20 releases for the same storage, more if we don't support every python version.

  • Tag new release. This should trigger the upload to PyPI, but I might need some help if it fails (ie, to manually put some uploads up)

One option would be to leave one wheel per platform of 0.3.8 for now (which?)

@cookpa
Copy link
Member Author

cookpa commented Aug 6, 2024

image

image

Most recent pypi stats. Mostly Linux, less older python now. I could clean up some 0.3.8 packages and try a new release if it's OK with everyone

@ntustison
Copy link
Member

ntustison commented Aug 6, 2024 via email

@cookpa
Copy link
Member Author

cookpa commented Aug 7, 2024

Deleted 0.3.8, figure we may as well make more space and not deal with this again for longer.

I'm ready to tag v0.5.3. Do I need to do anything else before releasing? I presume after I tag, I should edit pyproject.toml and ants/__init__.py, and increment the version to 0.5.4?

@cookpa
Copy link
Member Author

cookpa commented Aug 7, 2024

It worked! But the workflow only sends Linux wheels. Is there some other mechanism to upload the other platforms?

@stnava
Copy link
Member

stnava commented Aug 7, 2024

yes - it's via twine --- this is from the ANTsPyMM README:

rm -r -f build/ antspymm.egg-info/ dist/
python3 -m  build .
python3 -m pip install --upgrade twine
python3 -m twine upload --repository antspymm dist/*

you need to add some stuff to pypirc ie a secret token created on pypi .... I automated this for antspymm which uploads upon a new tag.

@cookpa
Copy link
Member Author

cookpa commented Aug 7, 2024

OK thanks I will try it out

@cookpa
Copy link
Member Author

cookpa commented Aug 7, 2024

I uploaded a source dist, which works, and a bunch of wheels from the Github build. Annoyingly, pip won't install the wheel for my Mac unless I do

SYSTEM_VERSION_COMPAT=0 pip install antspyx

This has long been the case for me with Github wheels. I think I tried to fix this before but will take another look. Hopefully we can get wheels building correctly for the next release

@cookpa
Copy link
Member Author

cookpa commented Aug 7, 2024

I was able to make a compatible wheel with

export MACOSX_DEPLOYMENT_TARGET=10.14
python -m build --wheel

For some reason, my Intel Mac only lists 10_X as compatible targets, even though I have Mac OS 14. We might therefore want to change the wheels to target an earlier version, currently they target the runner OS (12 for Intel, 14 for ARM).

My next to-do will be to try building wheels for an earlier OS target.

@cookpa
Copy link
Member Author

cookpa commented Aug 12, 2024

#697 I documented these Mac issues and set the wheel workflow to push everything to PyPI.

I cannot get the GHA workflow to allow an earlier MACOSX_DEPLOYMENT_TARGET, it fails with an error message about libpng being targeted for the current OS. I also have Homebrew libpng, so I'm not sure why my machine is OK with the MACOSX_DEPLOYMENT_TARGET=10.14.

I only added a python 3.11 wheel by hand, but can add others if needed.

@cookpa cookpa closed this as completed Aug 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
distribution Issues with CI, PyPI, etc question
Projects
None yet
Development

No branches or pull requests

4 participants