Updating to v9 from v7 seems to buffer streaming OpenAI calls

[joakimriedel]

Library name and version

Azure.AI.OpenAI_1.0.0-beta.9

Describe the bug

I just migrated an existing streaming implementation from v7 to v9. All seems good, and I prefer the new API structure, but now I seem to get some kind of buffering happening. The stream won't start sending updates until the entire response has been received, afterwards it just quickly rushes through all updates of the stream to the client. This kind of defeats the idea of streaming 😄

I will try to debug this further but my code diff is such a simple drop in replacement, so I would guess the buffering occurs internally somehow in the OpenAI.GetChatCompletionsStreaming() method. I wanted to post this here in the meantime if someone else has the same issue after migration and could give me some pointers to where I should look for a solution.

Expected behavior

The update chunks should flow through the stream at the same pace as they are received from server side, as they did with v7.

Actual behavior

All update chunks start to stream after the entire response has been buffered.

Reproduction Steps

Follow the migration steps here #39347

Environment

.NET 7

[joakimriedel]

A few hours later it seems to work without any code changes.. Perhaps it was some kind of server-side error in Azure OpenAI ?

I'll get back to this issue if I see this behaviour reproducing again.