Certain passwords fail with Extra-index-url with password in pip.conf

[jmons]

Environment

• pip version: pip 19.2.3
• Python version: python 3.7
• OS: Mac osx 10.14.6

Description
When I have a ^ in my password (which was auto generated), then Pip throws an error when trying to look at the private pypi. This has started to happen on 'newer versions of pip' (unclear which version broke it).

• the browser, and curl, don't have problems when using it this password, i.e. curl https://jt:Pro:REDACT^@REDACT/simple/cryptography/ gives back the redirect to the right place ok.

My pip.conf looks like this:

[global]
timeout = 20
extra-index-url = https://jt:REDACTED^@REDACTED/simple

Expected behavior
Not an error but the intended package should be installed from the correct server.

How to Reproduce

1. Have a private pypi server,
2. create a user with a password with a ^ in it.
3. Add that pypi with the password to pip.conf
4. Install any package (especially a public one from normal pypi)
5. Look at errors.

Output

(venv) MacBook-Pro:REDACT/ jt$ pip install -r config/pip.list 
Looking in indexes: https://pypi.org/simple, https://jt:****@REDACTED/simple
Collecting cryptography>=2.1.1 (from -r config/pip.list (line 2))
ERROR: Could not install packages due to an EnvironmentError: Failed to parse: https://jt:Pro:REDACTED^@REDACT/simple/cryptography/

Also in this instance, the password is then printed in the error, which is handy (there was a previous feature which auto-redacts it from the 'looking in indexes' output.

I wonder if this is related to #6775 ?

[chrahunt]

Hello. The problem likely isn't with ^, which isn't a reserved character as far as I know, but with the : in your username.

Please see here for details. Essentially, your username should be encoded as jt%3APro when in a URL.

[jmons]

Sorry, there may have been some copy paste errors - im not sure where the Pro is coming from. the : is seperating the username from the password as in the url is :

https://username:passwordwitha^@servername/simple

I reset my password to both with and without the ^ on that end (not sure if it breaks if the ^ is in the middle of the password). Is it now that username/passwords now need to be url encoded where before they didn't? (I note that ^ isn't on the list of reserved characters in URL's, but its also not on the reserved list either)

[chrahunt]

The recent change in the issue you linked is that @ needs to be URL encoded when it is in the username field, when previously it did not.

You can try encoding with %5E, it looks like ^ can cause issues per here.

[jmons]

So adding %5e instead of a ^ definitely works. I think my query is that this is something that changed - that there are probably a whole group (although a minority) of users out there for whom it was working fine and then one day it just stops working. I'm not clear why it stoped working, as which bit of the parser or sender is causing problems?

I'm happy to close this one, but I wonder if the documentation could be updated - that said I couldn't find any documentations specifically about this - the extra-index-url documentation seems to say 'see index-url', and the index-url basically says 'use a url'. TBH, the entire putting your password in a url thing is bad (tm), and I think a push to phase out support for it is happening else where? (I would prefer long term if it could tie into keychain or something, but I realise that doesn't work for everyone).

[sfdye]

It's updated:
https://pip.pypa.io/en/stable/user_guide/#basic-authentication-credentials