pip receives invalid Content-Type header from PyPi get request

[hmstepanek]

Describe the bug
pip receives invalid Content-Type header from PyPi get request.

Example 1:

https://github.com/newrelic/newrelic-python-agent/runs/7621668567?check_suite_focus=true

WARNING: Skipping page https://pypi.org/simple/pytest/ because the GET request got Content-Type: Unknown. The only supported Content-Types are application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and text/html
ERROR: Could not find a version that satisfies the requirement pytest==6.2.5 (from versions: none)
ERROR: No matching distribution found for pytest==6.2.5

Example 2:

https://github.com/newrelic/newrelic-python-agent/runs/7621668309?check_suite_focus=true

WARNING: Skipping page https://pypi.org/simple/beautifulsoup4/ because the GET request got Content-Type: Unknown. The only supported Content-Types are application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and text/html
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of pytest to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement beautifulsoup4 (from webtest) (from versions: none)
ERROR: No matching distribution found for beautifulsoup4

We tried explicitly disabling the caching directory but this did not fix the issue. At around the same time this issue began, we noticed the following issue was filled in the pip repo: #11949. We also noticed that the following issue appears to be the same issue we are running into now though the root cause may be different: pypa/pip#5345.

Expected behavior
pip receives valid Content-Type header from PyPi get request and is able to install the package.

To Reproduce

1. Kick off a test run by clicking the refresh icon (labeled "Re-run this job") on this page.
   and continue to do so until the tests fail (sometimes it fails the first time, sometimes it takes 2 or 3 runs to fail).

**Note we have not been able to reproduce this locally on our machines, only inside our Github Actions CI.

Output

https://github.com/newrelic/newrelic-python-agent/runs/7634470208?check_suite_focus=true

[9046] /home/runner/work/newrelic-python-agent/newrelic-python-agent$ /home/runner/work/newrelic-python-agent/newrelic-python-agent/.tox/postgres-datastore_psycopg2-py39-psycopg20208/bin/pip install pytest==6.2.5 iniconfig pytest-cov WebTest==2.0.35 'psycopg2-binary<2.9'
WARNING: Skipping page https://pypi.org/simple/pytest/ because the GET request got Content-Type: Unknown. The only supported Content-Types are application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and text/html
ERROR: Could not find a version that satisfies the requirement pytest==6.2.5 (from versions: none)
ERROR: No matching distribution found for pytest==6.2.5
ERROR: invocation failed (exit code 1)
ERROR: could not install deps [pytest==6.2.5, iniconfig, pytest-cov, WebTest==2.0.35, psycopg2-binary<2.9]; v = InvocationError("/home/runner/work/newrelic-python-agent/newrelic-python-agent/.tox/postgres-datastore_psycopg2-py39-psycopg20208/bin/pip install pytest==6.2.5 iniconfig pytest-cov WebTest==2.0.35 'psycopg2-binary<2.9'", 1)
postgres-datastore_psycopg2-py39-psycopg20208 finish: getenv /home/runner/work/newrelic-python-agent/newrelic-python-agent/.tox/postgres-datastore_psycopg2-py39-psycopg20208 after 1.94 seconds
cleanup /home/runner/work/newrelic-python-agent/newrelic-python-agent/.tox/.tmp/package/2/newrelic-0.0.zip

=================================== log end ====================================
✖ FAIL postgres-datastore_psycopg2-py39-psycopg20208 in 4.419 seconds
postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207 uses /opt/hostedtoolcache/Python/3.6.15/x64/bin/python3.6
postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207 start: parallel postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207 
[9112] /home/runner/work/newrelic-python-agent/newrelic-python-agent$ /opt/hostedtoolcache/Python/3.10.5/x64/bin/python3.10 /opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/tox/__main__.py -vv -e postgres-datastore_asyncpg-py37,postgres-datastore_asyncpg-py39,postgres-datastore_postgresql-py36,postgres-datastore_postgresql-py38,postgres-datastore_psycopg2-py27-psycopg20208,postgres-datastore_psycopg2-py37-psycopg20208,postgres-datastore_psycopg2-py39-psycopg20208,postgres-datastore_psycopg2cffi-py27-psycopg2cffi0207,postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207,postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207,postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208,postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208 -p auto --installpkg .tox/.tmp/package/1/newrelic-0.0.zip >.tox/postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207/log/postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207-0.log
postgres-datastore_psycopg2cffi-py27-psycopg2cffi0207 finish: parallel postgres-datastore_psycopg2cffi-py27-psycopg2cffi0207  after 20.55 seconds
✔ OK postgres-datastore_psycopg2cffi-py27-psycopg2cffi0207 in 20.555 seconds
pypy (/opt/hostedtoolcache/PyPy/2.7.18/x64/bin/pypy) is {'executable': '/opt/hostedtoolcache/PyPy/2.7.18/x64/bin/pypy', 'implementation': 'PyPy', 'version_info': [2, 7, 18, 'final', 42], 'version': '2.7.18 (8e99af2f9b8e3a58dd8691378a36ef43c3139ee9, Mar 29 2022, 05:16:07)\n[PyPy 7.3.9 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]', 'is_64': True, 'sysplatform': 'linux2', 'os_sep': '/', 'extra_version_info': [7, 3, 9, 'final', 0]}
postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207 uses /opt/hostedtoolcache/PyPy/2.7.18/x64/bin/pypy
postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207 start: parallel postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207 
[9962] /home/runner/work/newrelic-python-agent/newrelic-python-agent$ /opt/hostedtoolcache/Python/3.10.5/x64/bin/python3.10 /opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/tox/__main__.py -vv -e postgres-datastore_asyncpg-py37,postgres-datastore_asyncpg-py39,postgres-datastore_postgresql-py36,postgres-datastore_postgresql-py38,postgres-datastore_psycopg2-py27-psycopg20208,postgres-datastore_psycopg2-py37-psycopg20208,postgres-datastore_psycopg2-py39-psycopg20208,postgres-datastore_psycopg2cffi-py27-psycopg2cffi0207,postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207,postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207,postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208,postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208 -p auto --installpkg .tox/.tmp/package/1/newrelic-0.0.zip >.tox/postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207/log/postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207-0.log
postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207 finish: parallel postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207  after 21.97 seconds
✔ OK postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207 in 21.973 seconds
postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208 uses /opt/hostedtoolcache/Python/3.7.13/x64/bin/python3.7
postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208 start: parallel postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208 
[10603] /home/runner/work/newrelic-python-agent/newrelic-python-agent$ /opt/hostedtoolcache/Python/3.10.5/x64/bin/python3.10 /opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/tox/__main__.py -vv -e postgres-datastore_asyncpg-py37,postgres-datastore_asyncpg-py39,postgres-datastore_postgresql-py36,postgres-datastore_postgresql-py38,postgres-datastore_psycopg2-py27-psycopg20208,postgres-datastore_psycopg2-py37-psycopg20208,postgres-datastore_psycopg2-py39-psycopg20208,postgres-datastore_psycopg2cffi-py27-psycopg2cffi0207,postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207,postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207,postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208,postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208 -p auto --installpkg .tox/.tmp/package/1/newrelic-0.0.zip >.tox/postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208/log/postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208-0.log
postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208 finish: parallel postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208  after 20.76 seconds
✔ OK postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208 in 20.764 seconds
postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208 uses /opt/hostedtoolcache/Python/3.9.13/x64/bin/python3.9
postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208 start: parallel postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208 
[11484] /home/runner/work/newrelic-python-agent/newrelic-python-agent$ /opt/hostedtoolcache/Python/3.10.5/x64/bin/python3.10 /opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/tox/__main__.py -vv -e postgres-datastore_asyncpg-py37,postgres-datastore_asyncpg-py39,postgres-datastore_postgresql-py36,postgres-datastore_postgresql-py38,postgres-datastore_psycopg2-py27-psycopg20208,postgres-datastore_psycopg2-py37-psycopg20208,postgres-datastore_psycopg2-py39-psycopg20208,postgres-datastore_psycopg2cffi-py27-psycopg2cffi0207,postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207,postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207,postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208,postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208 -p auto --installpkg .tox/.tmp/package/1/newrelic-0.0.zip >.tox/postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208/log/postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208-0.log
postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207 finish: parallel postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207  after 27.38 seconds
✔ OK postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207 in 29.762 seconds
postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208 finish: parallel postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208  after 20.10 seconds
✔ OK postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208 in 20.1 seconds
___________________________________ summary ____________________________________
  postgres-datastore_asyncpg-py37: commands succeeded
  postgres-datastore_asyncpg-py39: commands succeeded
  postgres-datastore_postgresql-py36: commands succeeded
  postgres-datastore_postgresql-py38: commands succeeded
  postgres-datastore_psycopg2-py27-psycopg20208: commands succeeded
  postgres-datastore_psycopg2-py37-psycopg20208: commands succeeded
ERROR:   postgres-datastore_psycopg2-py39-psycopg20208: parallel child exit code 1
  postgres-datastore_psycopg2cffi-py27-psycopg2cffi0207: commands succeeded
  postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207: commands succeeded
  postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207: commands succeeded
  postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208: commands succeeded
  postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208: commands succeeded
cleanup /home/runner/work/newrelic-python-agent/newrelic-python-agent/.tox/.tmp/package/1/newrelic-0.0.zip
Error: Process completed with exit code 1.

My Platform

pip version

22.2.1

Python version

I have personally seen failures on 3.9.13, 3.8.13, 2.7.18. These failures appear to be independent of the Python version.

OS

Ubuntu 20.04.4 LTS

Additional context
Link to issue opened against pip: pypa/pip#11340. They think this is a PyPi bug and an incorrect response from the server though it looks like they may be implementing a workaround.

[dstufft]

If someone is able to get a copy of the actual response they're receiving, both headers and body, that would be extremely helpful in diagnosing what's wrong here.

[di]

This seems to be a duplicate of #3826 and pypa/pip#5345, which should be resolved. Given that, I'm going to close this in favor of pypa/pip#11340 as PyPI will not produce a Content-Type: Unknown and this is likely an issue with the client.