requests sends HTTP auth details in Host header

[cool-RR]

I noticed that the Host header of my http requests generated by requests have the HTTP auth credentials, like this:

Host: admin:[email protected]

(This is when I specified them in the URL.)

Is this standard? It looks weird to me.

[Lukasa]

Wow, that's wrong! I don't have my laptop right now so I can't investigate, but can anyone repro with the latest version of requests?

[sigmavirus24]

>>> import requests
>>> r = requests.get('https://username:[email protected]/get')
>>> r.json()['headers']
{'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'python-requests/2.4.1 CPython/3.3.2 Darwin/13.3.0', 'Authorization': 'Basic dXNlcm5hbWU6Zm9v', 'X-Request-Id': 'fb06edf3-14c8-42ad-b3c5-fefa801ddd9a', 'Connection': 'close', 'Host': 'httpbin.org'}
>>> r.request.headers
{'Authorization': 'Basic dXNlcm5hbWU6Zm9v', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Connection': 'keep-alive', 'User-Agent': 'python-requests/2.4.1 CPython/3.3.2 Darwin/13.3.0'}

Note that this doesn't seem to be reproducable. @cool-RR can you share the code and requests version that caused this?

[cool-RR]

Ian, can you please check using something like Fiddler the true Host of the
request?

On Thu, Sep 25, 2014 at 6:44 PM, Ian Cordasco [email protected]
wrote:

import requests>>> r = requests.get('https://username:[email protected]/get')>>> r.json()['headers']{'Accept': '/', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'python-requests/2.4.1 CPython/3.3.2 Darwin/13.3.0', 'Authorization': 'Basic dXNlcm5hbWU6Zm9v', 'X-Request-Id': 'fb06edf3-14c8-42ad-b3c5-fefa801ddd9a', 'Connection': 'close', 'Host': 'httpbin.org'}>>> r.request.headers{'Authorization': 'Basic dXNlcm5hbWU6Zm9v', 'Accept': '/', 'Accept-Encoding': 'gzip, deflate', 'Connection': 'keep-alive', 'User-Agent': 'python-requests/2.4.1 CPython/3.3.2 Darwin/13.3.0'}

Note that this doesn't seem to be reproducable. @cool-RR
https://github.com/cool-RR can you share the code and requests version
that caused this?

—
Reply to this email directly or view it on GitHub
https://github.com/kennethreitz/requests/issues/2247#issuecomment-56838438
.

[sigmavirus24]

@cool-RR I'll do that when you can give me the version and the code. The true host received by httpbin is what is printed on the 4th line. Also, it would look as though we're stripping the auth out and creating an Authorization header as we should be, before we even pass the URL on to urllib3.

[cool-RR]

Python 2.7.6 (default, Nov 10 2013, 19:24:24) [MSC v.1500 64 bit (AMD64)]
on win32
Type "copyright", "credits" or "license()" for more information.
DreamPie 1.2.1

import requests
requests.version
0: '2.4.1'
requests.get('http://yo:[email protected]')
1: <Response [400]>

Request in Fiddler raw view:

GET http://yo:[email protected]/ HTTP/1.1
Host: yo:[email protected]
Connection: keep-alive
Accept: /
Accept-Encoding: gzip, deflate
Authorization: Basic eW86cHc=
User-Agent: python-requests/2.4.1 CPython/2.7.6 Windows/7

Also Fiddler shows another warning about the port not being specified. Also
should the credentials show in the first line too?

On Thu, Sep 25, 2014 at 6:47 PM, Ian Cordasco [email protected]
wrote:

@cool-RR https://github.com/cool-RR I'll do that when you can give me
the version and the code. The true host received by httpbin is what is
printed on the 4th line. Also, it would look as though we're stripping the
auth out and creating an Authorization header as we should be, before we
even pass the URL on to urllib3.

—
Reply to this email directly or view it on GitHub
https://github.com/kennethreitz/requests/issues/2247#issuecomment-56838948
.

[Lukasa]

This bug is quite clearly proxy specific. =) Our generating of Host headers and Request URLs is clearly wrong there.

[cool-RR]

Sorry, I'm confused. Why is this clearly proxy-specific?

On Thu, Sep 25, 2014 at 7:15 PM, Cory Benfield [email protected]
wrote:

This bug is quite clearly proxy specific. =) Our generating of Host
headers and Request URLs is clearly wrong there.

—
Reply to this email directly or view it on GitHub
https://github.com/kennethreitz/requests/issues/2247#issuecomment-56843117
.

[Lukasa]

The full request URL is only sent when sent to a proxy. I presume you have your $HTTP_PROXY environment variable set to Fiddler. We've detected that and used our proxy-specific handling. This is why @sigmavirus24 couldn't reproduce your behaviour.

[sigmavirus24]

I'm seeing this with mitmproxy (and using explicity proxy settings). If I use netcat then I see this:

So I can confirm this is a proxy-specific issue.

[cool-RR]

I suggest comparing the output from the proxy-specific request to the
normal one to find any other diversions (like maybe port number missing or
anything else)

On Thu, Sep 25, 2014 at 7:27 PM, Ian Cordasco [email protected]
wrote:

I'm seeing this with mitmproxy (and using explicity proxy settings). If I
use netcat then I see this:

[image: screen shot 2014-09-25 at 11 26 45 am]
https://cloud.githubusercontent.com/assets/240830/4408147/d623579a-44d0-11e4-9147-0a0a7b700612.png

So I can confirm this is a proxy-specific issue.

—
Reply to this email directly or view it on GitHub
https://github.com/kennethreitz/requests/issues/2247#issuecomment-56844956
.

[t-8ch]

I blame https://github.com/shazow/urllib3/blob/f28732e73e9af6656d32584221ccc8b4bdb83920/urllib3/poolmanager.py#L242
(Investigating, there was a bugreport about this in the past)

[Lukasa]

That explains why it's in the Host header, but not why it's in the request URL.

Both are likely to be my fault though, that's code I wrote.

[t-8ch]

Na, I was wrong, I mixed up the auth info and the port

[t-8ch]

https://hg.python.org/cpython/file/bfdb995e8d7d/Lib/http/client.py#l1061
urlsplit from the stdlib includes the auth stuff in the netloc.
We should probably remove the auth stuff from the url before passing the URL down to httplib.

[t-8ch]

We have to remove the auth info here:
https://github.com/kennethreitz/requests/blob/a718a81d273503bd2ffae8e6cb036a8516eb426a/requests/adapters.py#L273

[shazow]

Did we decide if/where the bug was in urllib3? I'd like to have it fixed in urllib3 also.

[t-8ch]

I assumed, urllib3 expected users to not include auth stuff in URLs but set them in the headers

[shazow]

Mmm, I guess that's one interpretation. Not sure I'd expect that, at least with proxy_from_url.

But either way, I'm still confused about how auth was getting into Url.netloc? Was it all because of non-:// schemas?

[t-8ch]

It was in the URL which was passed as is down to httplib which used it in the request line and Host header

[t-8ch]

To clarify: It was never in Url.netloc, this was a mistake on my part. Urllib3 never touches the URL in the codepath used by requests

[shazow]

Ah k, thanks.

[theKashyap]

Probably redundant, but here's an example where I faced the problem: http://stackoverflow.com/questions/26109264/pip-proxy-authentication-and-not-supported-proxy-scheme