Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ChatEndpoint): implement TODO Streaming endpoints #42

Closed
XRSPACEHugo opened this issue Mar 8, 2023 · 2 comments · Fixed by #43
Closed

feat(ChatEndpoint): implement TODO Streaming endpoints #42

XRSPACEHugo opened this issue Mar 8, 2023 · 2 comments · Fixed by #43
Labels
enhancement New feature or request good first issue Good for newcomers

Comments

@XRSPACEHugo
Copy link

XRSPACEHugo commented Mar 8, 2023

public async Task StreamCompletionAsync(Action<ChatResponse> resultHandler, ChatRequest chatRequest, CancellationToken cancellationToken = default)
        {
            chatRequest.Stream = true;
            var jsonContent = JsonConvert.SerializeObject(chatRequest, Api.JsonSerializationOptions);
            using var request = new HttpRequestMessage(HttpMethod.Post, $"{GetEndpoint()}/completions")
            {
                Content = jsonContent.ToJsonStringContent()
            };
            var response = await Api.Client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, cancellationToken);
            await response.CheckResponseAsync(cancellationToken);
            await using var stream = await response.Content.ReadAsStreamAsync();
            using var reader = new StreamReader(stream);

            while (await reader.ReadLineAsync() is { } line)
            {
                if (line.StartsWith("data: "))
                {
                    line = line["data: ".Length..];
                }

                if (line == "[DONE]")
                {
                    return;
                }

                if (!string.IsNullOrWhiteSpace(line))
                {
                    resultHandler(DeserializeResult(response, line.Trim()));
                }
            }
        }

        private ChatResponse DeserializeResult(HttpResponseMessage response, string json)
        {
            var result = JsonConvert.DeserializeObject<ChatResponse>(json, Api.JsonSerializationOptions);

            if (result?.FirstChoice == null)
            {
                throw new HttpRequestException($"{nameof(DeserializeResult)} no completions! HTTP status code: {response.StatusCode}. Response body: {json}");
            }

            result.SetResponseData(response.Headers);

            return result;
        }
@XRSPACEHugo XRSPACEHugo added the enhancement New feature or request label Mar 8, 2023
@XRSPACEHugo
Copy link
Author

And make ChatResponse inherit BaseResponse :)

@XRSPACEHugo
Copy link
Author

BTW, I found
https://api.openai.com/v1/chat/completions will give us wrong data structure,
According to the document it will send back "message" in "choice"

ChatResponse

but we will get "delta" instead of "message" and make it deserialize wrong

@StephenHodgson StephenHodgson added the good first issue Good for newcomers label Mar 8, 2023
StephenHodgson added a commit that referenced this issue Mar 9, 2023
- refactored model validation
- added additional default models
- deprecate `OpenAIClient.DefaultModel`
- closes #42
github-actions bot pushed a commit that referenced this issue Mar 9, 2023
- refactored model validation
- added additional default models
- deprecate `OpenAIClient.DefaultModel`
- closes #42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers
Development

Successfully merging a pull request may close this issue.

2 participants