The pip-compile manager fails when compiling multiple inputs to one lock file and only one input includes private package registries

[mbudnek]

What would you like help with?

I think I found a bug

How are you running Renovate?

Mend Renovate hosted app on github.com

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

No response

Please tell us more about your question or problem

Pip-compile can compile multiple input files to one lock file. Renovate runs the manager's updateArtifacts function once for every input file, but currently it only looks up package registry credentials for input file passed. That means that given the following, Renovate will run the same pip-compile command twice, once with the necessary private registry credentials and once without:

A.in

--extra-index-url https://private.repo.com/pypi

private-package==1.2.3

B.in

public-package==2.0.0

requirements.txt

#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile --output-file=requirements.txt A.in B.in
#
--extra-index-url https://private.repo.com/pypi

private-package==1.2.3
    # via -r A.in
public-package==2.0.0
    # via -r B.in
# ...

When running updateArtifacts for 'A.in', Renovate will see the --extra-index-url flag and look up private.repo.com in its hostRules and pass the credentials to pip-compile, but when compiling 'B.in' it will run the exact same pip-compile command, but since 'B.in' has no --extra-index-url flag it won't pass credentials and pip-compile will fail to resolve 'private-package'.

As a short-term solution, the pip-compile manager's updateArtifacts function should look for --extra-index-url flags in all of the input files in the lock file's header.

As a longer-term solution, the pip-compile manager should use managerData to track its state and avoid running the exact same pip-compile command multiple times for the same set of input files.

Logs (if relevant)

Logs

[mbudnek]

I opened #28959 to provide the short-term solution I mentioned.