CDN Purging is broken, stale JSON responses are served instead

[di]

In #11920 we removed traversal for our legacy JSON views, however our purging mechanisms depend on traversal to determine when to issue purges. As a result, PyPI is currently serving stale data for all JSON endpoints for which responses existed before the merge of #11920, as no purges have been issued for these when something changes which would update the responses here (e.g., publishing a new release).

(ref: pypi/infra#104, #11936, #11937, #11939, #11940)

[gregorybchris]

Hi @di! Thanks for your work resolving this. I'm coming from the related python-poetry/poetry#3543 and mrabarnett/mrab-regex#475 issues. Will you confirm whether purging is finished for "projects which had old releases but new files added after the bug was introduced"? Thanks

[di]

Yes, should be. If you're still having problems though, let us know.

[emcd]

Still seeing problems. For example:
https://pypi.org/project/virtualenv/
Stated release is 20.16.0, but shield shows latest as 20.15.1.
curl shows the same behavior that I reported in #11936 several days ago:

$ curl --silent --location --header 'Accept: application/json' https://pypi.org/pypi/virtualenv/json | jq '.releases' | grep '20\.15\.1' || echo $?
  "20.15.1": [
      "filename": "virtualenv-20.15.1-py2.py3-none-any.whl",
      "url": "https://files.pythonhosted.org/packages/6f/43/df7c7b1b7a5ac4e41fac24c3682c1cc32f2c1d683d308bba2500338d1e3e/virtualenv-20.15.1-py2.py3-none-any.whl",
      "filename": "virtualenv-20.15.1.tar.gz",
      "url": "https://files.pythonhosted.org/packages/a4/2f/05b77cb73501c01963de2cef343839f0803b64aab4d5476771ae303b97a6/virtualenv-20.15.1.tar.gz",
$ curl --silent --location --header 'Accept: application/json' https://pypi.org/pypi/virtualenv/json | jq '.releases' | grep '20\.16' || echo $?
1

[gregorybchris]

I'm not as familiar with the metadata APIs, but I get Unable to find installation candidates for regex (2022.7.25), an error message from poetry, even after starting from scratch.

[emcd]

I'm not as familiar with the metadata APIs, but I get Unable to find installation candidates for regex (2022.7.25), an error message from poetry, even after starting from scratch.

@gregorybchris : Interesting. The JSON API shows a ton of wheels available for that release. (I won't paste the lengthy results list here, but you can see for yourself with something like: curl --silent --location --header 'Accept: application/json' https://pypi.org/pypi/regex/json | jq '.releases' | grep '2022\.7\.25') (You can also use the Simple API to see the list as well: curl --silent --location --header 'Accept: application/vnd.pypi.simple.v1+json' https://pypi.org/simple/regex | jq '.' | grep '2022\.7\.25') Some of the other people, who filed issues around the same time I did, reported that they could not find the release files on pythonhosted / pypi.io. Do you know which wheel Poetry is trying to fetch? Feels like there may actually be two different modes of breakage at work.

[neersighted]

I'm not as familiar with the metadata APIs, but I get Unable to find installation candidates for regex (2022.7.25), an error message from poetry, even after starting from scratch.

Have you cleared your poetry cache? poetry cache clear pypi --all

[gregorybchris]

Thanks @neersighted!! That worked.

[MatthewThe]

I'm still experiencing issues: https://pypi.org/pypi/picked-group-fdr/json still shows version 0.2.2, even though https://pypi.org/pypi/picked-group-fdr/0.2.3/json correctly shows the newest version

Edit:
The problem has now been resolved for my package.

[VeArnold]

Same issue as @MatthewThe.

https://pypi.org/pypi/codemagic-cli-tools/json shows 0.29.2 however the following links are up to date and existing:

https://pypi.org/pypi/codemagic-cli-tools/0.30.0/json
https://pypi.org/project/codemagic-cli-tools/0.30.0/

Edit:
Seems like it has been updated now, and we're able to update to 0.30.0. However, it took 1 hr 20 minutes to reach this point. Which is quite a bit of time, I reckon.

[nniehoff]

Same issue here. I am seeing the same issue with https://pypi.org/pypi/nautobot/json shows 1.3.8 however https://pypi.org/pypi/nautobot/1.3.9/json and https://pypi.org/project/nautobot/1.3.9/ show 1.3.9 has been released.

[bjornsnoen]

I released a new version of minicapy 20 hours ago which is still not visible in the API when requesting with headers Accept-Encoding: gzip,deflate, seems this issue was closed a bit too fast.

[adam-bloom]

@di dbt-core is having this issue for me as well - the json response still shows 1.1.1 even though 1.2.0 was released yesterday. Is this expected? I experimented locally, leaving off the Accept-Encoding header, and that got me 1.2.0, but requests adds the encoding header by default.

[dalthviz]

Hi, just in case, this is happening for jsonschema too (version 4.8.0 was released +-10 hours ago but the json response returns as latest release 4.7.2)

[emcd]

For people still encountering this issue, @dstufft recommended the following in #11936 to help PyPA debug:

It would be useful to know for someone who is getting a stale response, can they add the Fastly-Debug: 1 header, and see if there is a Surrogate-Key header, and if so, what the value is of that header?

[emcd]

Can confirm jsonschema is still broken:

$ curl --silent --location --header 'Accept: application/json' https://pypi.org/pypi/jsonschema/json | jq '.releases' | grep '4\.8\.0' || echo $?
1

$ curl --head --verbose --location --header 'Accept: application/json' --header 'Fastly-Debug: 1' https://pypi.org/pypi/jsonschema/json | egrep '(surrogate|fastly-debug|pypi|etag)'
etag: "MSWJBwrgR1jE+dkxcgGFlw"
surrogate-control: max-age=86400, stale-while-revalidate=300, stale-if-error=86400
surrogate-key:
x-pypi-last-serial: 14406929
fastly-debug-path: (D cache-bfi-krnt7300063-BFI 1659063631) (F cache-bfi-krnt7300080-BFI 1658588462) (D cache-iad-kjyo7100124-IAD 1658588462) (F cache-iad-kjyo7100124-IAD 1658550841)
fastly-debug-ttl: (H cache-bfi-krnt7300063-BFI - - 424) (H cache-iad-kjyo7100124-IAD 48778.980 86400.000 37621)
fastly-debug-digest: c2b71dfdc359c26a52aa1c6f362af62292fdbc19166c67968435e503b1e77686

[dalthviz]

Just in case, jsonschema is now working :) 🎉