Live stream to rtmps fails with mbedtls 3.6.0

[viric]

Operating System Info

Other

Other OS

NixOS (linux)

OBS Studio Version

30.2.0

OBS Studio Version (Other)

No response

OBS Studio Log URL

https://obsproject.com/logs/xSZCns85O2FeI8Mb

OBS Studio Crash Log URL

No response

Expected Behavior

OBS can send live stream to rtmps servers.

Current Behavior

OBS fails to connect to rtmps servers due to failure in TLS handshake.
It fails with my own server (peertube, that uses nodejs' tls module) and also, as an example, with facebook's rtmps server.

Steps to Reproduce

1. Configure a rtmps server with TLS1.3 as transmission output for live stream
2. Try to emit live stream

Anything else we should know?

I'm using mbedtls 3.6.0.
I had to patch obs to over come a requirement of the latest mbedtls: Mbed-TLS/mbedtls#9072
So I called psa_crypto_init() in RTMP_TLS_Init and made the TLS handshake to move further, but still fails.

I opened a ticket in mbedtls to see if they respond, adding there some of their debug info: Mbed-TLS/mbedtls#9415

I hope that giving the facebook rtmps host allows some progress.

[tytan652]

This is not an issue in OBS Studio but on MbedTLS.
They changed a behavior in a minor version that caused a breakage it's technically up to them to fix this breakage.

[viric]

I opened it here because they claim, at least, that psa_crypto_init() should be called now. But it's true that it still fails for other reasons.

[Fenrirthviti]

We are currently targeting 3.4.1 as our supported version. Incompatibility with 3.6 if they made a breaking change seems expected. If they are not interested in ensuring backwards compatibility, that's their choice.

This will likely be looked at when we update deps in the future, but I don't have any timeframe for when we might be looking to update mbedtls.

[viric]

AH I see that they will fix even the need of calling psa_crypto_init(): Mbed-TLS/mbedtls#9223
So that requires no change in obs.

[RytoEX]

For posterity, we are currently at an impasse with updating MbedTLS due to:

Building SRT 1.5.x against mbedTLS 3.5.0+ fails because of changes in mbedTLS that needed addressed in libsrt, but there is no new libsrt tag with the needed changes.
Haivision/srt#2842