RESTful sensors fail to update from HTTPS servers that do not support secure renegotiation since 2023.5.0 when underling OS is using openssl 3+

[trvrnrth]

The problem

Since updating to 2023.5.0 (running under Home Assistant Operating System) updates to sensors which consume from a server that does not support secure renogiation fail.

I do not have any control over the external server so cannot make it conformant.

What version of Home Assistant Core has the issue?

core-2023.5.0

What was the last working version of Home Assistant Core?

core-2023.4.6

What type of installation are you running?

Home Assistant OS

Integration causing the issue

RESTful

Link to integration documentation on our website

https://www.home-assistant.io/integrations/rest/

Diagnostics information

No response

Example YAML snippet

  - platform: rest
    name: Bin Data
    resource: https://www.bathnes.gov.uk/REDACTED
    scan_interval: 21600
    json_attributes:
      - residualNextDate
      - recyclingNextDate
      - organicNextDate
    value_template: "OK"

Anything in the logs that might be useful for us?

Logger: homeassistant.components.rest.sensor
Source: components/rest/sensor.py:72
Integration: RESTful
First occurred: 13:32:15 (1 occurrences)
Last logged: 13:32:15
Error connecting https://REDACTED failed with [SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disabled (_ssl.c:1007)

Additional information

It might make sense to add the ability to allow this on a per-resource basis. I believe the option required on the SSL context is as follows:

ssl_context.options |= 0x4 # set OP_LEGACY_SERVER_CONNECT

[home-assistant]

Hey there @epenet, mind taking a look at this issue as it has been labeled with an integration (rest) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of rest can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Renames the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign rest Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

---

rest documentation
rest source
_{^{(message by IssueLinks)}}

[epenet]

Can you please share your config, in particular verify_ssl and ssl_cipher_list? (redacting sensitive information)

cc @mib1185 who worked recently on adding some SSL options.

[trvrnrth]

@epenet I've added an example config snippet to the description (scraping bin collection dates from my council website). I've got SSL verification turned on (the default) and am also using the default cipher list.

For this use case I'd be happy enough not doing verification FWIW but as I understand it the issue at play relates to RFC 5746 and mitigation of it in the latest versions of Python/OpenSSL.

[mib1185]

unsafe legacy renegotiation has been disabled with OpenSSL 3.0.0 (see openssl/openssl@72d2670)
not sure which dependency bump in 2023.5 bumps openssl to 3.0.0 (maybe #91528)

[trvrnrth]

@mib1185 I'm not sure quite how the release pipeline works overall but I suspect it came in with the bump to alpine 3.17 in home-assistant/docker#268

Would it be possible to add an option to the integration to allow the behaviour? It feels like that would be the safest option as it seems sensible to make it opt-in only on a case-by-case basis.

[mib1185]

You're right, alpine 3.17 switched to openssl 3.0.
So it is not related to the rest integration itself and should also affect other integrations.
We need to discuss, if this should be solved central in HA core.

[bdraco]

They decided not to back port the option in cpython

python/cpython#89051

elk won't be upgrading their SSL support any time soon so the solution for that problem is here gwww/elkm1#69

[epenet]

I guess the question is: how do we make this available to the user in REST?
Does this become a fourth ssl_cipher_list? Does it become a new standalone config option?

[trvrnrth]

From the user perspective I'd initially imagined a new ssl_allow_legacy_server_connect config option defaulting to false would work nicely enough.

[mib1185]

this is the plan ... will work on it next few days

[mib1185]

@trvrnrth could you please provide the server hostname (no need for the whole URL) so I can test the new option during development

[trvrnrth]

@mib1185 Sure. I didn't really need to redact that bit. It's https://www.bathnes.gov.uk

[trozman]

Not sure if it's related, but my Rest sensor (MELCloud, heat pump API), which worked perfectly, also stopped working after the HA update (to HA 2023.5.2, supervisor: 2023.04.).
Log warning: REST result could not be parsed as JSON.

URL:
https://app.melcloud.com/Mitsubishi.Wifi.Client/user/ListDevices?id=XXXX&buildingID=YYYYY (xxx and yyy are replaced with my HP specific info)

[trvrnrth]

@trozman That will likely be something different. You can see the servers backing that endpoint support secure renegotiation in these SSL labs scan results.

[liranwaiss]

hi any new on this problem ? still
Error connecting https://api.ims.gov.il/v1/envista/stations/54/data/7/latest failed with [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1007)

[Tonguc-Endem]

@Mattwmaster58, as per your workaround suggestion, issuing the command pip install cryptography==36.0.2 results with error error: command 'gcc' failed: No such file or directory since I have HA OS.

[Tonguc-Endem]

I have the same problem.

I need to issue the following curl command but I get:
curl: (35) OpenSSL/3.1.0: error:0A000152:SSL routines::unsafe legacy renegotiation disabled

here is my command:

curl 'https://www.epdk.gov.tr/Detay/GetFastAccessList' \
  -H 'Accept: application/json, text/javascript, */*; q=0.01' \
  -H 'Accept-Language: en-US,en;q=0.9,tr;q=0.8' \
  -H 'Connection: keep-alive' \
  -H 'Content-Type: application/json; charset=UTF-8' \
  -H 'DNT: 1' \
  -H 'Origin: https://www.epdk.gov.tr' \
  -H 'Referer: https://www.epdk.gov.tr/Detay/Icerik/3-1327/elektrik-faturalarina-esas-tarife-tablolari' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Site: same-origin' \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36' \
  -H 'X-Requested-With: XMLHttpRequest' \
  -H 'sec-ch-ua: "Google Chrome";v="113", "Chromium";v="113", "Not-A.Brand";v="24"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "macOS"' \
  --data-raw '{"fId":"1327"}' \
  --compressed | jq '.model[0].RgDate'

the parsed result should be "31.03.2023"

This command is working fine on my other computers.

[Tonguc-Endem]

@mib1185 could you give a new milestone?

[cdevrell]

@trvrnrth Coincidently I have just tried to do the exact same thing using the BANES website and found this because I had the same error. As a (hopefully short-term) workaround until the BANES website is updated or a HA change is implemented, I have created a Python script to fetch the data and push to home assistant using a HTTP sensor instead. https://github.com/cdevrell/BinDayChecker

These lines in the script workaround the problem:

ssl_ctx = create_urllib3_context()
ssl_ctx.load_default_certs()
ssl_ctx.options |= 0x4

with urllib3.PoolManager(ssl_context=ssl_ctx) as http:
    response = http.request("GET", URL)

[trvrnrth]

@cdevrell In case it's of interest to you (or anyone else who stumbles across this with particular reference to BANES bin collection) I contributed mampfes/hacs_waste_collection_schedule#1015 and have been using that integration in the meantime.

[program-the-brain-not-the-heartbeat]

any update on this?

[codyc1515]

Still an issue.

[phismith91]

Any update on this? Is it working again? Got the same problem but I'm not sure if it's because of HA or my technical skills ;)

[codyc1515]

It became an issue after a HA update but the actual root cause is the HTTPS server is insecure.

[phismith91]

So still an issue on your side right? Any workarounds?

[codyc1515]

No, the HTTPS server needs upgrading. Run the website through the check here and post the result - https://www.ssllabs.com/ssltest/

[phismith91]

Ahh okay my fault. Missed the detail about the external url :/. Faced the same issue with my KNX server, then I have to check my config again. Thanks for your fast reply!

[codyc1515]

Is this solved?

[issue-triage-workflows]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.