fix: Guard against removing package while uploading to it #64
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
matthewfeickert
commented
Feb 15, 2024
matthewfeickert
temporarily deployed
to
remove-old-wheels
GitHub Actions
Inactive
matthewfeickert
temporarily deployed
to
remove-old-wheels
GitHub Actions
Inactive
matthewfeickert
temporarily deployed
to
remove-old-wheels
GitHub Actions
Inactive
matthewfeickert
temporarily deployed
to
remove-old-wheels
GitHub Actions
Inactive
matthewfeickert
commented
Feb 15, 2024
matthewfeickert
temporarily deployed
to
remove-old-wheels
GitHub Actions
Inactive
matthewfeickert
force-pushed
the
fix/check-if-removing-will-delete-package
branch
from
ab1b478 to
f87a5b5
Compare
matthewfeickert
temporarily deployed
to
remove-old-wheels
GitHub Actions
Inactive
|
@kenodegard @dholth as you know the Anaconda Cloud web APIs better than I do, if you would have time to review the changes to |
|
@scientific-python/nightly-wheels-developers If it would be possible to get a review early this week it would be great to get this in so that I can create a new patch release with this. (I'm assuming when I tag @scientific-python/nightly-wheels-developers that everyone on the team gets a notification except me (the tagger), but I'm not sure if that's correct or not.) |
matthewfeickert
commented
Feb 19, 2024
* On Anaconda Cloud as of 2024-01-04, if a wheel is being uploaded to a
package, but the package only has one wheel in it and is of the same
name as the uploaded wheel, Anaconda Cloud will overwrite the file by
_removing_ the file from the package index. However, when this happens
it removes the entire package, and then the wheel that is in the
process of being uploaded has no destination and the upload fails.
To guard against this, ensure for each package that has a wheel being
uploaded if:
- there is only one release for the package
- and only 1 file for that release
- and the upload target wheel has the same name as the file
- that the file (and so the package) is removed in advance of the
upload.
* To make filtering names and versions from wheels easier, add a
get_wheel_name_version function that uses as regex to lazily capture
the package name as well as the version and then return these.
- Examples of this working:
* "matplotlib-3.9.0.dev0-pp39-pypy39_pp73-win_amd64.whl"
matplotlib 3.9.0.dev0
* "scikit_learn-1.5.dev0-cp39-cp39-win_amd64.whl"
scikit_learn 1.5.dev0
* "scipy-openblas64-0.3.26.186-py3-none-macosx_10_9_x86_64.whl"
scipy-openblas64 0.3.26.186
* "awkward_cpp-29-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
awkward_cpp 29
* As Anaconda Cloud normalizes package names, also distinguish between
basename and package name to try to make things easier to keep track
of when normalizing.
* Example:
- basename: test_package-0.0.1-py3-none-any.whl
- basename prefix: test_package
- package_name: test-package
matthewfeickert
force-pushed
the
fix/check-if-removing-will-delete-package
branch
from
f87a5b5 to
c4a9b40
Compare
matthewfeickert
temporarily deployed
to
remove-old-wheels
GitHub Actions
Inactive
|
@scientific-python/nightly-wheels-developers Ping again on this, as given that apparently merging now requires a minimum of 1 review this can't get merged until someone reviews. |
jarrodmillman
approved these changes
Feb 21, 2024
|
Thanks @jarrodmillman! I'm going to relock, merge, and then prep a new minor release as this add behavior without introducing breaking changes. |
* As curl and jq are now used in cmd.sh, they need to also be added to the conda environment.yml. * The lower bounds are chosen as the latest values, but are not motivated by known problems. * Rebuild the lock file.
* Add a test that triggers the conditions for removal of a package from Anaconda Cloud in advance of its upload to avoid an error.
matthewfeickert
force-pushed
the
fix/check-if-removing-will-delete-package
branch
from
c4a9b40 to
c37220f
Compare
matthewfeickert
temporarily deployed
to
remove-old-wheels
GitHub Actions
Inactive
matthewfeickert
deleted the
fix/check-if-removing-will-delete-package
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #57
On Anaconda Cloud as of 2024-01-04, if a wheel is being uploaded to a package, but the package only has one wheel in it and is of the same name as the uploaded wheel, Anaconda Cloud will overwrite the file by removing the file from the package index. However, when this happens it removes the entire package, and then the wheel that is in the process of being uploaded has no destination and the upload fails. To guard against this, ensure for each package that has a wheel being uploaded if:
upload.
To make filtering names and versions from wheels easier, add a
get_wheel_name_versionfunction that uses as regex to lazily capture the package name as well as the version and then return these."matplotlib-3.9.0.dev0-pp39-pypy39_pp73-win_amd64.whl"matplotlib3.9.0.dev0"scikit_learn-1.5.dev0-cp39-cp39-win_amd64.whl"scikit_learn1.5.dev0"scipy-openblas64-0.3.26.186-py3-none-macosx_10_9_x86_64.whl"scipy-openblas640.3.26.186"awkward_cpp-29-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"awkward_cpp29"awkward-2.6.1-py3-none-any.whl"awkward2.6.1As this requires
curlandjqalso add these to the environment and relock.Add a extra upload in CI to thest this behavior.
In the future there will hopefully be a fix on Anaconda Cloud's side given anaconda/anaconda-client#702 but we need a solution now.