Incorrect behavior with schemeless-dotless host:port URLs #6455

itamaro · 2023-05-13T21:19:31Z

URLs of the form hostname:8080 (with no scheme, with "hostname" not containing any dots) can be used to refer to the netloc "hostname:8080"

requests.utils.prepend_scheme_if_needed should correctly prepend the new_scheme when provided with such a URL.

Expected Result

the prepended-scheme URL should be "http://hostname:8080"

Actual Result

the prepended-scheme URL is "hostname:///8080" (e.g. treating the "hostname" part as the scheme, no host, no port, and "8080" as the path)

I extended the test_prepend_scheme_if_needed to demonstrate this behavior (see main...itamaro:requests:schemeless-hostname-anad-port-bug)

Reproduction Steps

from requests.utils import prepend_scheme_if_needed
print(prepend_scheme_if_needed("hostname:8080", "http"))

System Information

$ python -m requests.help

{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "3.1.0"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.4"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.10.4"
  },
  "platform": {
    "release": "22.4.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.30.0"
  },
  "system_ssl": {
    "version": "101010ef"
  },
  "urllib3": {
    "version": "2.0.2"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}

The text was updated successfully, but these errors were encountered:

sigmavirus24 · 2023-05-14T00:50:51Z

How does this realistically affect users who aren't using utils which aren't part of the public API?

itamaro · 2023-05-14T02:31:32Z

How does this realistically affect users who aren't using utils which aren't part of the public API?

ah sorry, I went down a rabbit hole tracking down the issue to this that I forgot to mention the user-facing scenario!

one scenario is when using such a URL as a proxy:

response = requests.get(
    "http://www.example.com",
    proxies={"http": "myproxy:8080"},
    ....
)

this used to work (with requests 2.25.1 and python 3.8), but with requests 2.27.1 it fails with

...
requests.exceptions.InvalidProxyURL: Please check proxy URL. It is malformed and could be missing the host.

another scenario (less critical) is difference in exceptions (MissingSchema vs InvalidSchema):

response = requests.get("www.example.com")
...
MissingSchema: Invalid URL 'www.example.com': No scheme supplied. Perhaps you meant http://www.example.com?

vs

response = requests.get("hostname:8080")
...
requests.exceptions.InvalidSchema: No connection adapters were found for 'hostname:8080'

sigmavirus24 · 2023-05-14T10:44:47Z

So the last two aren't ever supposed to work. Would it be nice if they raised the same exception? Sure. But never should we be guessing scheme. We never document that schemeless URLs are supported in that way.

The proxy case may take investigation. But I suspect we documented a breaking change there. I vaguely remember other people complaining

itamaro · 2023-05-15T00:23:21Z

The proxy case may take investigation. But I suspect we documented a breaking change there. I vaguely remember other people complaining

Yes, it's definitely the proxy case that brought me here and is causing me issues. The other one is just a nit.

nateprewitt · 2023-05-15T17:39:03Z

Hi @itamaro, this was an intentional change in 2.27.0 due to this bug in CPython. The behavior of urlparse fundamentally changed from Python 3.9 onwards which makes supporting schemeless URIs in this case fairly difficult. The decision was made to move to urllib3's url_parse function, which is the behavior you're seeing now to limit blast radius with the the standard library updates.

This is why you're seeing an error now that wasn't occurring previously.

Python 3.10

Python 3.10.5 (main, Jul  1 2022, 17:28:53) [Clang 13.0.0 (clang-1300.0.27.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from urllib.parse import urlparse
>>> urlparse('hostname:8080')
ParseResult(scheme='hostname', netloc='', path='8080', params='', query='', fragment='')

Python 3.7

Python 3.7.9 (default, Aug 11 2022, 16:47:29) 
[Clang 13.0.0 (clang-1300.0.27.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from urllib.parse import urlparse
>>> urlparse('hostname:8080')
ParseResult(scheme='', netloc='', path='hostname:8080', params='', query='', fragment='')
>>>

The only reason this happened to work previously is we did this (arguably bad) shuffle of replacing the netloc with the path for this case. That honestly should have never been done but was there to work around the oddities of url parsing in the standard library.

So the short of the story is all version of Requests will be affected by this as soon as you upgrade beyond Python 3.8. We have no reliable way to control this from Requests side anymore. I would recommend updating your usage to ensure proxies are passed with a scheme to avoid any surprises later.

Requests 2.25.1 behavior on Python 3.10

>>> response = requests.get(
...     "http://www.example.com",
...     proxies={"http": "myproxy:8080"},
...)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/nateprewitt/.pyenv/versions/3.10.5/lib/python3.10/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/Users/nateprewitt/.pyenv/versions/3.10.5/lib/python3.10/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/nateprewitt/.pyenv/versions/3.10.5/lib/python3.10/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/nateprewitt/.pyenv/versions/3.10.5/lib/python3.10/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/Users/nateprewitt/.pyenv/versions/3.10.5/lib/python3.10/site-packages/requests/adapters.py", line 412, in send
    conn = self.get_connection(request.url, proxies)
  File "/Users/nateprewitt/.pyenv/versions/3.10.5/lib/python3.10/site-packages/requests/adapters.py", line 309, in get_connection
    proxy_manager = self.proxy_manager_for(proxy)
  File "/Users/nateprewitt/.pyenv/versions/3.10.5/lib/python3.10/site-packages/requests/adapters.py", line 193, in proxy_manager_for
    manager = self.proxy_manager[proxy] = proxy_from_url(
  File "/Users/nateprewitt/.pyenv/versions/3.10.5/lib/python3.10/site-packages/urllib3/poolmanager.py", line 492, in proxy_from_url
    return ProxyManager(proxy_url=url, **kw)
  File "/Users/nateprewitt/.pyenv/versions/3.10.5/lib/python3.10/site-packages/urllib3/poolmanager.py", line 429, in __init__
    raise ProxySchemeUnknown(proxy.scheme)
urllib3.exceptions.ProxySchemeUnknown: Not supported proxy scheme myproxy

itamaro · 2023-05-16T00:06:42Z

thanks for the background @nateprewitt !

I would recommend updating your usage to ensure proxies are passed with a scheme to avoid any surprises later.

totally agree this is the preferred solution, but it's not going to be easy doing that in our monorepo with millions of lines of code... (e.g. the entire Meta Python codebase 😬)

turingnixstyx · 2023-05-17T22:38:48Z

Guys should I try solving this or is this redundant?

nateprewitt · 2023-05-17T22:42:27Z

@turingnixstyx I don't think there's anything to be solved. Requests cannot support schemeless proxies in Python 3.9+. It's an unfortunately tedious change for end-users, but we can't do much about the behavior in the standard library. What we're doing now is providing a consistent behavior across all Python versions.

The urlparse stdlib call behaves differently on Python 3.9+ for schemaless-URLs. It parses string hostname into a schema instead of netloc. The issue is still open and discussed * psf/requests#6455 and apparently it will not be solved, so we need to workaround it if we still want to support legacy, schemaless URLs to be accepted by Elasticsearch handler.

The urlparse stdlib call behaves differently on Python 3.9+ for schemaless-URLs. It parses string hostname into a schema instead of netloc. The issue is still open and discussed * psf/requests#6455 and apparently it will not be solved, so we need to workaround it if we still want to support legacy, schemaless URLs to be accepted by Elasticsearch handler. GitOrigin-RevId: dd73a0bffa6c4de93a2dd8dc4460b64aedc51255

This comment was marked as off-topic.

Sign in to view

potiuk mentioned this issue Aug 10, 2023

Fix urlparse schemaless-behaviour on Python 3.9+ apache/airflow#33289

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incorrect behavior with schemeless-dotless host:port URLs #6455

Incorrect behavior with schemeless-dotless host:port URLs #6455

itamaro commented May 13, 2023

sigmavirus24 commented May 14, 2023

itamaro commented May 14, 2023

sigmavirus24 commented May 14, 2023

itamaro commented May 15, 2023

nateprewitt commented May 15, 2023

itamaro commented May 16, 2023

turingnixstyx commented May 17, 2023

nateprewitt commented May 17, 2023 •

edited

Loading

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

Incorrect behavior with schemeless-dotless host:port URLs #6455

Incorrect behavior with schemeless-dotless host:port URLs #6455

Comments

itamaro commented May 13, 2023

Expected Result

Actual Result

Reproduction Steps

System Information

sigmavirus24 commented May 14, 2023

itamaro commented May 14, 2023

sigmavirus24 commented May 14, 2023

itamaro commented May 15, 2023

nateprewitt commented May 15, 2023

itamaro commented May 16, 2023

turingnixstyx commented May 17, 2023

nateprewitt commented May 17, 2023 • edited Loading

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

nateprewitt commented May 17, 2023 •

edited

Loading