Include linehaul information in user agent

[pradyunsg]

This is effectively a request to include download-related information available to PyPI when interacting with the index server, so that informtion can used to make ecosystem-wide decision (by querying said information via https://warehouse.pypa.io/api-reference/bigquery-datasets.html#download-statistics-table).

https://github.com/pypi/linehaul-cloud-function is the PyPI side implementation. https://github.com/pypa/pip/blob/24.0/src/pip/_internal/network/session.py#L109 is the pip side implementation.

This data powers decision making such as https://pypistats.org/packages/__all__ (and similar sites), https://mayeut.github.io/manylinux-timeline/ and a few ad-hoc queries to determine usage patterns across the ecosystem.

[charliermarsh]

Perfect, thanks @pradyunsg!

[charliermarsh]

Everything here looks pretty straightforward (and we should already have it all available), perhaps with the exception of setuptools_version.

[pradyunsg]

perhaps with the exception of setuptools_version

Yea, skipping values you don't have readily available seems fine.

[hauntsaninja]

Last I benchmarked, construction of user agent was surprisingly expensive in pip, took like 200ms.

[hauntsaninja]

Hm, looks like it might cost up to 600ms in a representative environment:

λ hyperfine -w 3 'python -c "import pkg_resources; from pip._internal.network.session import *; user_agent()"' 'python -c "import pkg_resources; from pip._internal.network.session import *"'
Benchmark 1: python -c "import pkg_resources; from pip._internal.network.session import *; user_agent()"
  Time (mean ± σ):      1.423 s ±  0.031 s    [User: 0.677 s, System: 0.365 s]
  Range (min … max):    1.388 s …  1.482 s    10 runs
 
Benchmark 2: python -c "import pkg_resources; from pip._internal.network.session import *"
  Time (mean ± σ):     854.2 ms ±   6.0 ms    [User: 414.9 ms, System: 198.2 ms]
  Range (min … max):   843.4 ms … 862.7 ms    10 runs

If pip were to skip getting setuptools version then pip's user agent construction costs 190ms for me. Next most expensive thing is rustc version, which wouldn't be horrible to cache. I guess off topic for this tracker, but Pradyun let me know if pip is interested in PRs here

[konstin]

I've asked upstream about rustc startup time: rust-lang/rustup#2626 (comment). Is there a page i can link with rust version stats to motivate this use case?

Next most expensive thing is rustc version, which wouldn't be horrible to cache.

The entrypoint is a shim, so afaik you can't cache it reliably (i'm happy to query the default rustc from a different place though).

Is the setuptools information from uv relevant given that we don't install it be default, and always use the latest (compatible) version in build envs?

Except for the rustc call and the (base) interpreter metadata call we cache already, i think we should be able to do without any subprocess calls, i.e. fast enough it's not noticeable on profiles.

[alex]

In general I would say that the setuptools information is much less useful than it used to be: projects who need a newer setuptools can now reliably depend on it with build-system.requires.

What do you mean a page with rust version stats? Are you asking where you can see the stats from PyPI? They're all available in BigQuery. Here's an example of the type of analysis I do with it:
pyca_cryptography_rage_dashboard (1).pdf

[charliermarsh]

Let’s just stick to what we have access to (I’d like to omit rustc and setuptools for now).

[konstin]

What do you mean a page with rust version stats? Are you asking where you can see the stats from PyPI? They're all available in BigQuery. Here's an example of the type of analysis I do with it:
pyca_cryptography_rage_dashboard (1).pdf

Yes, something like all but including the rust version. Getting the data from bigquery is quite the overhead if someone wants just simple caniuse.com style check.

[alex]

Ah. I'm not aware of any website that displays rust versions for all of PyPI.