Which install flags should pip use?

[vidartf]

After #5386, I've had some issues when installing my project with pip on top of a conda environment. The error messages from pip that I'm seeing are along the lines of this:

Cannot uninstall 'certifi'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

This is caused by the following conditions:

• The --force-install flag to pip causes all dependencies to be reinstalled as well.
• Some quite common, deeper dependencies (docutils, certifi, etc) are missing metadata for pip to uninstall when installed with conda. See e.g. Upgrading to pip 10: It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall. pypa/pip#5247 for why pip >= 10 complains about this.

It is not necessarily clear why the --force-install/--ignore-installed flag was there initially (see comments in #5386), but it might be an issue of caching or other optimizations.

A possible solution to this would be to expose some configuration options which determines which flag(s) to use, if any, but I'm not sure what the threshold is for adding options to the configuration?

[vidartf]

Possible flags to consider:

• --ignore-installed: What was there before Fix pip installs #5386. See the PR for arguments against it.
• --force-install: Gives the issues explained here (and might increase install times since it reinstalls all deps).
• --upgrade: From CLI help:

  Upgrade all specified packages to the newest available version. The handling of dependencies depends on the upgrade-strategy used.

  • --upgrade-strategy <upgrade_strategy>:

    Determines how dependency upgrading should be handled [default: only-if-needed]. "eager" - dependencies are upgraded regardless of whether the currently installed version satisfies the requirements of the upgraded package(s). "only-if-needed" - are upgraded only when they do not satisfy the requirements of the upgraded package(s).

• No flags.

[vidartf]

For my case, I think no flags would be sufficient:

• It is installing the project package from a path. As far as I can tell from looking at pip, this means it will always be reinstalled (?).
• Dependencies will only be (re-)installed if current versions are conflicting with requirements. As I'm using conda, most deps will be recent / up to date. This ensure they are neither stale, and that pip is unlikely to try to reinstall them (avoiding the issue reported in the initial report).

However, an approach that might be a better approximation of what we have now could be to use --upgrade --upgrade-strategy eager:

• It will always reinstall the project package.
• Dependencies will eagerly be updated to the latest version. This is unlikely to cause the error I'm seeing, as long as the conda and pip distributions are reasonably in sync. It will also prevent dependencies from being stale.

[stsewd]

Thanks for digging in to this! I read the thread and seems that a solution is downgrading pip pypa/pip#5247 (comment)

So, maybe you can install that version of pip from conda? Another option will be having a requirements.txt installed with that version of pip https://docs.readthedocs.io/en/stable/config-file/v2.html#python-install

Sadly, we don't support installing requirements when using conda :/ #2776

[mgeier]

I had the same problem with this readthedocs.yml:

conda:
    file: doc/readthedocs-environment.yml
python:
    pip_install: true

This is the error log: https://readthedocs.org/projects/nbsphinx/builds/8920771/

It worked again when I changed to setup_py_install:

python:
    setup_py_install: true

[stsewd]

Is this fixed with #5635?

[chrisjsewell]

Is this fixed with #5635?

For me yes, its working fine now. As I mention in the PR, perhaps a more comprehensive solution may be to include additional options in the .readthedocs.yaml, to control the flags used when pip installing. But I'll leave that for you to decide, if it is worth the effort 😀

[vidartf]

Should be ok in terms of this issue, but if somebody else has a similar issue, I would recommend them to comment on this issue to evaluate if further changes are needed 👍

[stsewd]

As I mention in the PR, perhaps a more comprehensive solution may be to include additional options in the .readthedocs.yaml, to control the flags used when pip installing

@agjohnson any thoughts here?

I'm closing this one, but if needed we should open a new one proposing to use the config file to define what flags to use in pip. Kind of related with #3986