pip-compile fails with pip==22.0

[jordan-dimov]

I upgraded Pip to latest version 22.0 today and pip-tools stopped working.

When I run pip-compile to upgrade my requirements, I now get the following stack trace:

$ pip-compile -U requirements/requirements.in
Traceback (most recent call last):
  File "/Users/jordan/.virtualenvs/my-project/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
  File "/Users/jordan/.virtualenvs/my-project/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/jordan/.virtualenvs/my-project/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/jordan/.virtualenvs/my-project/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/jordan/.virtualenvs/my-project/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/jordan/.virtualenvs/my-project/lib/python3.7/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/jordan/.virtualenvs/my-project/lib/python3.7/site-packages/piptools/scripts/compile.py", line 342, in cli
    repository = PyPIRepository(pip_args, cache_dir=cache_dir)
  File "/Users/jordan/.virtualenvs/my-project/lib/python3.7/site-packages/piptools/repositories/pypi.py", line 106, in __init__
    self._setup_logging()
  File "/Users/jordan/.virtualenvs/my-project/lib/python3.7/site-packages/piptools/repositories/pypi.py", line 455, in _setup_logging
    assert isinstance(handler, logging.StreamHandler)
AssertionError

As soon as I downgrade pip to v. 21.3.1 for example, the above command succeeds as expected.

Environment Versions

1. OS Type: MacOS
2. Python version: 3.7.10
3. pip version: 22.0
4. pip-tools version: 6.4.0

[hugovk]

pypa/pip#10726 (comment):

That looks like a bug on their end -- they're using pip's internals (something that is not supported) and quite possibly they weren't prepared for pypa/pip#10461. :)

[fsecada01]

I pushed a merge request that fixes the issue for me, but it's failing tests.

#1559

[lsaavedr]

this is a big bug 🤯

[iflare3g]

I pushed a merge request that fixes the issue for me, but it's failing tests.

#1559
I left a comment as there's a typo, try to re-check if tests will pass after fixing it

Thanks for the merge request 🙏🏻

[zbentley]

The workaround, for folks affected, is pip install pip<22 in the environment where you're using pip-tools.

[RAbraham]

I think we may have to do more than pip install pip<22? I just tried that but my next command to install pip-tools downloads pip 22 again?

pip install "pip<22"


pip install pip-tools


Collecting pip>=21.2
  Downloading pip-22.0.2-py3-none-any.whl (2.1 MB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/site-packages (from pip-tools) (57.0.0)
Collecting click>=7
  Downloading click-8.0.3-py3-none-any.whl (97 kB)
Requirement already satisfied: wheel in /usr/local/lib/python3.7/site-packages (from pip-tools) (0.36.2)
Collecting pep517
  Downloading pep517-0.12.0-py2.py3-none-any.whl (19 kB)
Collecting importlib-metadata
  Using cached importlib_metadata-4.10.1-py3-none-any.whl (17 kB)
Collecting typing-extensions>=3.6.4
  Using cached typing_extensions-4.0.1-py3-none-any.whl (22 kB)
Collecting zipp>=0.5
  Using cached zipp-3.7.0-py3-none-any.whl (5.3 kB)
Collecting tomli>=1.1.0
  Downloading tomli-2.0.0-py3-none-any.whl (12 kB)
Installing collected packages: zipp, typing-extensions, tomli, importlib-metadata, pip, pep517, click, pip-tools
  Attempting uninstall: pip
    Found existing installation: pip 21.1.3
    Uninstalling pip-21.1.3:
      Successfully uninstalled pip-21.1.3
Successfully installed click-8.0.3 importlib-metadata-4.10.1 pep517-0.12.0 pip-22.0.2 pip-tools-6.4.0 tomli-2.0.0 typing-extensions-4.0.1 zipp-3.7.0

[RAbraham]

Thanks!

…

________________________________ From: Stefan Baack ***@***.***> Sent: Monday, January 31, 2022 12:02 PM To: jazzband/pip-tools ***@***.***> Cc: Rajiv Abraham ***@***.***>; Comment ***@***.***> Subject: Re: [jazzband/pip-tools] pip-compile fails with pip==22.0 (Issue #1558) I think we may have to do more than pip install pip<22? I just tried that but my next command to install pip-tools downloads pip 22 again? You can add pip < 22 to your requirements.in file (or to setup.cfg/pyproject.toml if you use that instead) to prevent pip-tools from reinstalling the latest pip version. Just make sure you remove that line again once this issue here is fixed — Reply to this email directly, view it on GitHub<#1558 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AABUJLO3VB2NY7RDIXYXNALUY26BFANCNFSM5NERFCWQ>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[antonagestam]

@RAbraham Or as commands, you need to either install pip-tools first and then downgrade pip, e.g. pip install pip-tools; pip install 'pip<22'. Or just provide the limitation in the same command pip install pip-tools 'pip<22'. But yes, it probably makes more sense to put the limitations into your dependency spec file.

[rsalmaso]

It seems that with pip>=22 pip-tools removes dash normalization https://pip.pypa.io/en/stable/news/#id73 even if it was done in pip==21.2 🙄

[SethMMorton]

A similar occurrence happened when pip 21.3 was released last year: #1503.

On it there was some mention of making sure the pip-tools and pip maintainers stay in sync when breaking changes are made. Since it seems that pip is pretty liberal about refactoring their internals (which, to be clear, is their prerogative since it is internal) so that extra communication with pip-tools could help.

I wonder if this could be alleviated by moving the pip-tools project under the pypa umbrella. One of the pypa's most important products uses pip-tools so I would imagine they would see value in ensuring it stays in sync with pip. (As a side note, the fact that warehouse uses pip-tools and not pipenv is interesting, to say the least, considering that pipenv is the officially recommended tool by the pypa...). This suggestion is mirroring the sentiment here.

[createdbykartik]

I was at pip version 21.0.1. Upgraded to 21.2 and re-ran docker-compose, worked fine!

[hugovk]

This repo does have a daily cron to test against pip main, which is really good.

https://github.com/jazzband/pip-tools/actions/workflows/cron.yml

However, the Python 3.6 job started failing three months ago when pip dropped support for 3.6:

Which masked all the jobs started failing two months ago due to other internal pip changes:

https://github.com/jazzband/pip-tools/runs/4195776918?check_suite_focus=true

https://github.com/jazzband/pip-tools/runs/4419708428?check_suite_focus=true

---

I'm not sure whether a change in organisation would necessarily help, the pip team will continue to change their private internal functions as they need to (and there will be more changes to use Rich more for better output).

Wherever it lives, I'd suggest a closer eye is paid to the integration of pip with pip-tools, especially in the lead up to pip's quarterly releases. (And look out for the pip release issues for planned release dates e.g. pypa/pip#10726.)

[davisagli]

It looks like it should be possible to adjust the code that is poking at pip's logging internals to skip it when running against a version of pip that uses Rich -- it appears to be a workaround for an issue with progress bars (#949) that does not occur with the Rich-based output.

[hugovk]

Please see #1567 for a fix.

[sbaack]

I wonder if this could be alleviated by moving the pip-tools project under the pypa umbrella.

Nope. The fundamental problem here is that pip does not provide any public API -- see https://pip.pypa.io/en/stable/user_guide/#using-pip-from-your-program. pip-tools is one of the few projects in the wild that still uses pip's internal code as if it's an API and this sort of breakage is exactly what the risk of that usage pattern is.

Just to add my 2 cents, I wish pip would add a compile and sync flag to effectively integrate pip-tools's functionality into standard pip. Then issues like this one here won't arise in stable pip releases and we would have good dependency management out of the box. Probably not going to happen, but one can dream :-)

[chrihartl]

I updated pip and pip-tools to the latest versions (22.0.3 and 6.5.1, respectively). Then I tried to run pip-compile:

pip-compile -vvv -U --no-emit-index-url requirements.in

But there is absolutely no output (also no error message), and the file requirements.txt does not get created (nor updated, if it exists).

When I downgrade pip-tools to 6.4.0 (but keep pip at version 22) I get an AssertionError.

It seems that the only way to fix my problem is to downgrade pip to version 21.3. Wasn't pip-tools>=6.5.0 supposed to fix (or work around) the (pip>=22) issue?

EDIT: I managed to get pip-compile working with the latest versions of pip and pip-tools, but this required me to perform the following tasks in the given order:

python -m pip install "pip<22"
pip uninstall pip-tools
pip install -U pip-tools
python -m pip install -U pip