PEP 691: JSON-based simple index API interaction #11158

dstufft · 2022-06-01T06:07:41Z

No description provided.

src/pip/_internal/index/collector.py

uranusjr · 2022-06-01T06:18:12Z

Makes sense to me from a quick read.

pfmoore · 2022-06-01T08:59:08Z

LGTM. It's nice how simple this change is as well 🙂

dstufft · 2022-06-01T18:57:19Z

I did just test this with EpicWink/proxpi#8 as well, since that's easier to use against live PyPI data instead of Warehouse's development data. It was using JSON and it worked like a charm.

dstufft · 2022-06-15T04:01:34Z

Note: I think this is slightly bugged, it's dropping the hash data on the floor, as I think the Link class depends on it being in the URL.

dstufft · 2022-06-25T18:33:43Z

Testing this:

$ pip -vvv download requests --no-deps
...
Fetched page https://pypi.org/simple/requests/ as application/vnd.pypi.simple.v1+json
...

dstufft · 2022-06-26T16:25:11Z

I'm not going to merge this myself, as I don't feel confident in my knowledge of the code base anymore to give myself the 👍, but I believe this should be good to go now if someone wants to review it.

pfmoore

I've only desk-checked this but it LGTM. I'll leave it for now in case anyone else has any comments, but if no-one else comes up with any objections, I'll merge it.

pradyunsg · 2022-06-26T21:43:26Z

Looks like the PyPI implementation landed, so it should be possibel to use that to try this out. :)

fschulze · 2022-06-27T05:10:01Z

I added PEP 691 support in devpi-server, both for fetching from PyPI and serving installers.

The instance at https://m.devpi.net/ runs the 6.6.0 development version with the added support. For testing with pip you could use -i https://m.devpi.net/root/pypi/+simple/.

An added benefit of the Accept header approach is easy and performant support for other forms of the URL without the trailing slash or the +simple, before we did User-Agent based detection for installers to prevent redirects or serving the devpi-web interface.

You can get the wheels of the development releases for local testing here:
https://m.devpi.net/fschulze/dev/+f/220/2b013a4eadc7f/devpi_common-3.6.1.dev0-py2.py3-none-any.whl#sha256=2202b013a4eadc7f0d41528058db9416c8eab7ddd552fe032c6a4ddfd80c5b5b
https://m.devpi.net/fschulze/dev/+f/a65/b973cb95dd00c/devpi_server-6.6.0.dev0-py3-none-any.whl

dstufft · 2022-07-06T23:52:52Z

Seems like nobody seems to have any concerns, I think this should be good to land now?

sbidoul

LGTM (code review only, no manual test).

Since this will switch to using the new API without transition, is there anything particular to mention in the release notes ?

src/pip/_internal/index/collector.py

src/pip/_internal/models/link.py

src/pip/_internal/index/collector.py

sbidoul · 2022-07-15T09:33:52Z

@dstufft I plan to release 22.2 in a week or so. Will you have time to look at the comments above by then -- I'm aware you are super busy on the PyPI side - huges thanks for that 🙏 ❤️ -- ?

dstufft · 2022-07-15T14:00:17Z

I missed those comments, yea I can address those.

sbidoul · 2022-07-16T08:59:16Z

Thanks, @dstufft ! Do you see anything we (or users) need to pay attention to on release day ?

dstufft · 2022-07-16T14:40:37Z

At the risk of jinxing things, this should be entirely transparent.

notatallshaw · 2022-07-16T14:51:13Z

At the risk of jinxing things, this should be entirely transparent.

I think my only concern would be because you are changing the accept headers it's possible that the software that is being used for workflows inside organizations that are using private repos might give unexpected results.

Unfortunately I no longer work for such an organization otherwise I'd be happy to test it. But something to keep an eye out during the release process.

fschulze · 2022-07-16T15:08:25Z

@notatallshaw any default configuration should handle the accept headers just fine. With devpi-server it definitely works.

sbidoul · 2022-07-16T17:17:55Z

We could imagine an index doing a non-standard compliant parsing of the Accept header to test for text/html.

Anyone knows if we can give them a heads up ?

In the meantime I'm going ahead and merge this to move forward in the release process.

Bumps [pip](https://github.com/pypa/pip) from 22.1.2 to 22.2. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pypa/pip/blob/main/NEWS.rst">pip's changelog</a>.</em></p> <blockquote> <h1>22.2 (2022-07-21)</h1> <h2>Deprecations and Removals</h2> <ul> <li>Remove the <code>html5lib</code> deprecated feature flag. (<code>[#10825](pypa/pip#10825) <https://github.com/pypa/pip/issues/10825></code>_)</li> <li>Remove <code>--use-deprecated=backtrack-on-build-failures</code>. (<code>[#11241](pypa/pip#11241) <https://github.com/pypa/pip/issues/11241></code>_)</li> </ul> <h2>Features</h2> <ul> <li> <p>Add support to use <code>truststore <https://pypi.org/project/truststore/></code>_ as an alternative SSL certificate verification backend. The backend can be enabled on Python 3.10 and later by installing <code>truststore</code> into the environment, and adding the <code>--use-feature=truststore</code> flag to various pip commands.</p> <p><code>truststore</code> differs from the current default verification backend (provided by <code>certifi</code>) in it uses the operating system’s trust store, which can be better controlled and augmented to better support non-standard certificates. Depending on feedback, pip may switch to this as the default certificate verification backend in the future. (<code>[#11082](pypa/pip#11082) <https://github.com/pypa/pip/issues/11082></code>_)</p> </li> <li> <p>Add <code>--dry-run</code> option to <code>pip install</code>, to let it print what it would install but not actually change anything in the target environment. (<code>[#11096](pypa/pip#11096) <https://github.com/pypa/pip/issues/11096></code>_)</p> </li> <li> <p>Record in wheel cache entries the URL of the original artifact that was downloaded to build the cached wheels. The record is named <code>origin.json</code> and uses the PEP 610 Direct URL format. (<code>[#11137](pypa/pip#11137) <https://github.com/pypa/pip/issues/11137></code>_)</p> </li> <li> <p>Support <code>PEP 691 <https://peps.python.org/pep-0691/></code><em>. (<code>[#11158](pypa/pip#11158) <https://github.com/pypa/pip/issues/11158></code></em>)</p> </li> <li> <p>pip's deprecation warnings now subclass the built-in <code>DeprecationWarning</code>, and can be suppressed by running the Python interpreter with <code>-W ignore::DeprecationWarning</code>. (<code>[#11225](pypa/pip#11225) <https://github.com/pypa/pip/issues/11225></code>_)</p> </li> <li> <p>Add <code>pip inspect</code> command to obtain the list of installed distributions and other information about the Python environment, in JSON format. (<code>[#11245](pypa/pip#11245) <https://github.com/pypa/pip/issues/11245></code>_)</p> </li> <li> <p>Significantly speed up isolated environment creation, by using the same sources for pip instead of creating a standalone installation for each environment. (<code>[#11257](pypa/pip#11257) <https://github.com/pypa/pip/issues/11257></code>_)</p> </li> <li> <p>Add an experimental <code>--report</code> option to the install command to generate a JSON report of what was installed. In combination with <code>--dry-run</code> and <code>--ignore-installed</code> it can be used to resolve the requirements. (<code>[#53](pypa/pip#53) <https://github.com/pypa/pip/issues/53></code>_)</p> </li> </ul> <h2>Bug Fixes</h2> <ul> <li>Fix <code>pip install --pre</code> for packages with pre-release build dependencies defined both in <code>pyproject.toml</code>'s <code>build-system.requires</code> and <code>setup.py</code>'s <code>setup_requires</code>. (<code>[#10222](pypa/pip#10222) <https://github.com/pypa/pip/issues/10222></code>_)</li> <li>When pip rewrites the shebang line in a script during wheel installation, update the hash and size in the corresponding <code>RECORD</code> file entry. (<code>[#10744](pypa/pip#10744) <https://github.com/pypa/pip/issues/10744></code>_)</li> <li>Do not consider a <code>.dist-info</code> directory found inside a wheel-like zip file as metadata for an installed distribution. A package in a wheel is (by</li> </ul>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pypa/pip/commit/8e7e76e60f4e115ea1201bee2f176377a718fce1"><code>8e7e76e</code></a> Bump for release</li> <li><a href="https://github.com/pypa/pip/commit/b6f6a94e36f10a4535ea5bbdc6b351f62003eede"><code>b6f6a94</code></a> Update AUTHORS.txt</li> <li><a href="https://github.com/pypa/pip/commit/790725aca3f60c745e33827a6079d9600da373d8"><code>790725a</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11274">#11274</a> from sbidoul/install-report-note-sbi</li> <li><a href="https://github.com/pypa/pip/commit/d4b9e187aa7cc5ab14b2339f6171f7f2ea6504e9"><code>d4b9e18</code></a> Add clarifications to the installation report documentation</li> <li><a href="https://github.com/pypa/pip/commit/b1a01ef762a78af1194958a1c874015eaf81fd04"><code>b1a01ef</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11265">#11265</a> from finnagin/main</li> <li><a href="https://github.com/pypa/pip/commit/48bcb0a4ccd30a9d00e58fe58827772e307a7e39"><code>48bcb0a</code></a> reformat to pass pre-commit check</li> <li><a href="https://github.com/pypa/pip/commit/a7c1fe3bff5655393018c53b448b669b3525515b"><code>a7c1fe3</code></a> Remove utc fixture from tests</li> <li><a href="https://github.com/pypa/pip/commit/0c574f72905185d62bcca741c813df9bae1d9282"><code>0c574f7</code></a> Remove time import</li> <li><a href="https://github.com/pypa/pip/commit/246fef19149eea893f1cf3efd53f9b17c94c952f"><code>246fef1</code></a> Remove utc fixture</li> <li><a href="https://github.com/pypa/pip/commit/c9cb7f4629bdd8c61b792feff6dacb1d2e848d57"><code>c9cb7f4</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11270">#11270</a> from uranusjr/upgrade-pre-commit-hooks</li> <li>Additional commits viewable in <a href="https://github.com/pypa/pip/compare/22.1.2...22.2">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pip&package-manager=pip&previous-version=22.1.2&new-version=22.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details>

uranusjr reviewed Jun 1, 2022

View reviewed changes

src/pip/_internal/index/collector.py Outdated Show resolved Hide resolved

dstufft changed the title ~~PoC of PEP 691~~ PEP 691 Jun 25, 2022

dstufft force-pushed the pep691 branch from c779621 to f5ab3e8 Compare June 25, 2022 18:23

pradyunsg added type: enhancement Improvements to functionality PEP implementation Involves some PEP C: finder PackageFinder and index related code labels Jun 26, 2022

dstufft marked this pull request as ready for review June 26, 2022 13:49

pfmoore approved these changes Jun 26, 2022

View reviewed changes

pradyunsg changed the title ~~PEP 691~~ PEP 691: JSON-based simple index API interaction Jun 27, 2022

sbidoul added this to the 22.2 milestone Jul 7, 2022

sbidoul reviewed Jul 9, 2022

View reviewed changes

src/pip/_internal/index/collector.py Outdated Show resolved Hide resolved

src/pip/_internal/models/link.py Show resolved Hide resolved

src/pip/_internal/index/collector.py Show resolved Hide resolved

dstufft added 8 commits July 15, 2022 17:22

PoC of PEP 691

6f167b5

Support Hashes from PEP691

1fc571f

Use a tuple and one call to startswith

a287a0b

Use smaller values for HTML

9c58d2d

fix typing

21fadc5

Fix tests

55192a1

Add logging

63bd865

Fix Typing on older Pythons

5600eea

dstufft added 13 commits July 15, 2022 17:22

Fix import ordering

b64a067

Add a news entry

cdc2582

Update tests to use new names

21a99e4

Add new accept headers

9c98346

Update some tests to work again

799c88f

formatting

d67cac3

Fix link hashes

6f9ccfc

fix a stray '

7cb5aae

Always use Unknown when unknown

5f3fbf0

fix number of calls

68836d4

Better expand on our TODO

41051f5

Document the hashes parameter

c1b46c1

Add a test for parsing links from JSON

c1b50e1

dstufft force-pushed the pep691 branch from fa4217a to c1b50e1 Compare July 15, 2022 21:55

sbidoul merged commit dc00479 into pypa:main Jul 16, 2022

di mentioned this pull request Jul 25, 2022

Roadmap: integration into pip as pip audit subcommand pypa/pip-audit#335

Open

13 tasks

github-actions bot locked as resolved and limited conversation to collaborators Aug 2, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PEP 691: JSON-based simple index API interaction #11158

PEP 691: JSON-based simple index API interaction #11158

dstufft commented Jun 1, 2022

uranusjr commented Jun 1, 2022

pfmoore commented Jun 1, 2022

dstufft commented Jun 1, 2022

dstufft commented Jun 15, 2022

dstufft commented Jun 25, 2022

dstufft commented Jun 26, 2022

pfmoore left a comment

pradyunsg commented Jun 26, 2022

fschulze commented Jun 27, 2022 •

edited

Loading

dstufft commented Jul 6, 2022

sbidoul left a comment

sbidoul commented Jul 15, 2022

dstufft commented Jul 15, 2022

sbidoul commented Jul 16, 2022

dstufft commented Jul 16, 2022

notatallshaw commented Jul 16, 2022

fschulze commented Jul 16, 2022

sbidoul commented Jul 16, 2022 •

edited

Loading

PEP 691: JSON-based simple index API interaction #11158

PEP 691: JSON-based simple index API interaction #11158

Conversation

dstufft commented Jun 1, 2022

uranusjr commented Jun 1, 2022

pfmoore commented Jun 1, 2022

dstufft commented Jun 1, 2022

dstufft commented Jun 15, 2022

dstufft commented Jun 25, 2022

dstufft commented Jun 26, 2022

pfmoore left a comment

Choose a reason for hiding this comment

pradyunsg commented Jun 26, 2022

fschulze commented Jun 27, 2022 • edited Loading

dstufft commented Jul 6, 2022

sbidoul left a comment

Choose a reason for hiding this comment

sbidoul commented Jul 15, 2022

dstufft commented Jul 15, 2022

sbidoul commented Jul 16, 2022

dstufft commented Jul 16, 2022

notatallshaw commented Jul 16, 2022

fschulze commented Jul 16, 2022

sbidoul commented Jul 16, 2022 • edited Loading

fschulze commented Jun 27, 2022 •

edited

Loading

sbidoul commented Jul 16, 2022 •

edited

Loading