Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updating dependency requests to 2.32.2 will fail with Not supported URL scheme http+unix #1182

Closed
henkjan-van-wijk opened this issue May 21, 2024 · 9 comments
Assignees
Labels
bug external bug Bug on some other library

Comments

@henkjan-van-wijk
Copy link

henkjan-van-wijk commented May 21, 2024

Context

  • OS and version used: Ubuntu 22.04
  • Python version: 3.12.2
  • pip version: 24.0
  • list of installed packages:
    Package Version

annotated-types 0.7.0
azure-iot-device 2.13.0
certifi 2024.2.2
charset-normalizer 3.3.2
deprecation 2.1.0
idna 3.7
janus 1.0.0
packaging 24.0
paho-mqtt 1.6.1
pip 24.0
pydantic 2.7.1
pydantic_core 2.18.2
PySocks 1.7.1
requests 2.32.2
requests-unixsocket 0.3.0
typing_extensions 4.11.0
urllib3 1.26.18

Description of the issue

Today we updated the dependencies on a test iot edge module and saw that the module no longer could connect to the edgeHub using IoTHubModuleClient.create_from_edge_environment().

After some testing to check which changes were made, it seems to boil down to the requests package which has been updated May 20, 2024 to 2.32.0 with some subsequent updates to 2.32.2 today. But this latest fix still results in the same issue.

Explicitly downgrading to 2.31.0 fixes the issue for now.

Code sample exhibiting the issue

I retested it with the sample code from https://github.com/Azure/azure-iot-sdk-python/blob/main/samples/async-edge-scenarios/invoke_method_on_module.py with same issue.

Console log of the issue

Traceback (most recent call last):
  File "/app/.venv/lib/python3.12/site-packages/requests/adapters.py", line 555, in send
    conn = self.get_connection_with_tls_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/requests/adapters.py", line 411, in get_connection_with_tls_context
    conn = self.poolmanager.connection_from_host(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/urllib3/poolmanager.py", line 246, in connection_from_host
    return self.connection_from_context(request_context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/urllib3/poolmanager.py", line 258, in connection_from_context
    raise URLSchemeUnknown(scheme)
urllib3.exceptions.URLSchemeUnknown: Not supported URL scheme http+unix

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/./main.py", line 36, in <module>
    asyncio.run(main())
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/app/./main.py", line 16, in main
    module_client = IoTHubModuleClient.create_from_edge_environment()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/azure/iot/device/aio/patch_documentation.py", line 228, in create_from_edge_environment
    return super(IoTHubModuleClient, cls).create_from_edge_environment(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/azure/iot/device/iothub/abstract_clients.py", line 816, in create_from_edge_environment
    server_verification_cert = hsm.get_certificate()
                               ^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/azure/iot/device/iothub/edge_hsm.py", line 61, in get_certificate
    r = requests.get(
        ^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/requests_unixsocket/__init__.py", line 51, in get
    return request('get', url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/requests_unixsocket/__init__.py", line 46, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/requests/adapters.py", line 559, in send
    raise InvalidURL(e, request=request)
requests.exceptions.InvalidURL: Not supported URL scheme http+unix
@steffenbeermann
Copy link

Thank you very much! Had the same issue and could not find the reason. Downgrading the requests solved it for now.

@cartertinney
Copy link
Member

cartertinney commented May 22, 2024

Thanks for bringing this to our attention.

The requests and associated urllib dependencies (and their interaction) have often created issues with our package. In this case, they made a security fix that broke another dependency, requests_unixsocket.

Unfortunately, requests_unixsocket does not appear to be actively maintained at this time, so I'm not sure how viable it will be for it to be updated to fix the issue. We will explore some potential solutions here, and if necessary, we will update our library to restrict the dependency to a version that works.

In the meantime (and in general) I would recommend sticking with a specific version of requests and urllib3 that has worked for you prior - this is not the first time such updates have caused breakage, and it likely won't be the last.

@cartertinney cartertinney self-assigned this May 22, 2024
@cartertinney cartertinney added the external bug Bug on some other library label May 22, 2024
@epa095
Copy link

epa095 commented May 25, 2024

@cartertinney : I think it is reasonable to see this as not only an external bug, but also a problem with this projects requirements specification. In setup.py it is specified:

"requests>=2.20.0,<3.0.0",
"requests-unixsocket>=0.1.5,<1.0.0",

which means that this should install cleanly in any environment which satisfy these limits. It is of course impossible to actually guarantee that you can handle all combinations, but its reasonable to expect the dependency requirements to at least be correct for the latest versions you specify for all dependencies. And the fact is that azure-iot-sdk-python is incompatible with requests==2.32.*, and I think that the install_requires should reflect that.

So to sum up; I think you should also change the install_requires here, by either setting "requests>=2.20.0,<2.32.0", or something like "requests>=2.20.0,<3.0.0,!=2.32.*"

@cartertinney
Copy link
Member

@epa095

I agree, like I said, above:

We will explore some potential solutions here, and if necessary, we will update our library to restrict the dependency to a version that works.

I'm hesitant to come out and immediately restrict the version because the update to requests in question is a security fix - if we restrict our version in the long term, we're rejecting this security update as well as any future ones. If it comes down to it we will (at least in the short term), but that's not an ideal outcome either.

@fdelu
Copy link

fdelu commented May 29, 2024

There's an issue in the request-unixsocket repo for this issue: msabramo/requests-unixsocket#73

Since the project is abandoned an user forked the project with a fix. Would it be possible to either use that fork or for this SDK to have its own fork? The project is abandoned anyway so it shouldn't matter if the fork doesn't receive any updates other than critical bugs like this one

Edit: just noticed it's your issue @cartertinney. The comment still applies

@cartertinney
Copy link
Member

@fdelu

It's something we're considering, certainly, although it's not trivial for us to simply switch to the other fork, as we would need to do a security analysis, etc.

We're discussing our options right now, and will make some decisions on how to proceed in the coming days.

@fdelu
Copy link

fdelu commented May 29, 2024

I understand why just switching to the other fork could have security issues, that's why I suggested doing a separate fork just for this SDK and only fix critical bugs there.

I'll certainly wait for any updates in the next few days, thanks for the info!

@cartertinney
Copy link
Member

cartertinney commented Jun 5, 2024

Update: we will be simply restricting the requests version for the time being (see #1184).

A release will be made including these changes as soon as possible.

@cartertinney
Copy link
Member

azure-iot-device 2.13.1 has been released

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug external bug Bug on some other library
Projects
None yet
Development

No branches or pull requests

5 participants