websocket connection failed to Azure OpenAI realtime API

[daifuliu]

I tried to use this lib in a Unity app, to connect to Azure OpenAI gpt-4o-realtime-preview API, however I got this error:

WebSocketException: Unable to connect to the remote server
System.Net.WebSockets.WebSocketHandle.ParseAndValidateConnectResponseAsync (System.IO.Stream stream, System.Net.WebSockets.ClientWebSocketOptions options, System.String expectedSecWebSocketAccept, System.Threading.CancellationToken cancellationToken) (at <ebe52a74a56f408f9f7ffa692e89b0de>:0)
System.Net.WebSockets.WebSocketHandle.ConnectAsyncCore (System.Uri uri, System.Threading.CancellationToken cancellationToken, System.Net.WebSockets.ClientWebSocketOptions options) (at <ebe52a74a56f408f9f7ffa692e89b0de>:0)
System.Net.WebSockets.ClientWebSocket.ConnectAsyncCore (System.Uri uri, System.Threading.CancellationToken cancellationToken) (at <ebe52a74a56f408f9f7ffa692e89b0de>:0)
Utilities.WebSockets.WebSocket.ConnectAsync (System.Threading.CancellationToken cancellationToken) (at ./Library/PackageCache/[email protected]/Runtime/WebSocket.cs:167)
UnityEngine.Debug:LogException(Exception)
Utilities.WebSockets.<ConnectAsync>d__35:MoveNext() (at ./Library/PackageCache/[email protected]/Runtime/WebSocket.cs:214)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:SetResult(Int32)
Mono.Net.Security.<StartOperation>d__57:MoveNext()
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:SetResult(AsyncProtocolResult)
Mono.Net.Security.<StartOperation>d__23:MoveNext()
System.Runtime.CompilerServices.AsyncTaskMethodBuilder:SetResult()
Mono.Net.Security.<ProcessOperation>d__24:MoveNext()
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:SetResult(Nullable`1)
Mono.Net.Security.<InnerRead>d__25:MoveNext()
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:SetResult(Int32)
Mono.Net.Security.<InnerRead>d__66:MoveNext()
System.Threading._ThreadPoolWaitCallback:PerformWaitCallback()

Postman wss connection test to Azure endpoint was ok, wireshark logs showed tls handshake was ok but seemed failed at websocket upgrade,

any idea where the problem is or what to do next to debug further?
[Unity version: 2022.3.48f1c1]

[StephenHodgson]

I don't have a subscription to azure open ai, so PRs are welcome to fix this

[daifuliu]

Well, thanks much @StephenHodgson
MS provides free trial on the models, https://azure.microsoft.com/en-us/products/ai-services/openai-service/

[StephenHodgson]

Unfortunately I already used my trial.