Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add headers to response info #206

Open
Krobys opened this issue Jun 27, 2023 · 5 comments · May be fixed by #211
Open

Add headers to response info #206

Krobys opened this issue Jun 27, 2023 · 5 comments · May be fixed by #211
Labels
enhancement New feature or request

Comments

@Krobys
Copy link

Krobys commented Jun 27, 2023

Hello. This is a request for adding functionality. I need to have access to the fields in the headers that come with this API since there are some important fields there, for example:
x-ratelimit-limit-requests: 3500
x-ratelimit-limit-tokens: 180000
x-ratelimit-remaining-requests: 3499
x-ratelimit-remaining-tokens: 179972
x-ratelimit-reset-requests: 17ms
x-ratelimit-reset-tokens: 9ms

I need to have access to this headers data or some other headers. It would be great if you could add additional functions, for example, and they would return Pair<OUR REGULAR RESPONSE, HEADERS>. and will be located next to the usual functions. function1(): Response and function1Extra(): Pair<Response, Headers>.
@Krobys Krobys added the enhancement New feature or request label Jun 27, 2023
@aallam
Copy link
Owner

aallam commented Jun 28, 2023

This indeed can be a good enhancement, I will look into it soon!

@Krobys
Copy link
Author

Krobys commented Jun 30, 2023

played a bit with the code, here is possible way:

  1. add to interface Chat - function chatCompletionWithHeaders
  2. implement function in the ChatApi:
    override suspend fun chatCompletionWithHeaders(request: ChatCompletionRequest): Pair<ChatCompletion, Headers> {
    val httpResponse: HttpResponse = requester.perform {
    it.post {
    url(path = ApiPath.ChatCompletions)
    setBody(request)
    contentType(ContentType.Application.Json)
    }.body()
    }
    return httpResponse.body() to httpResponse.headers
    }

tested that and it works, but maybe you will find some other more abstract way

@Krobys Krobys linked a pull request Jul 12, 2023 that will close this issue
Open
@palhoye
Copy link

palhoye commented Dec 6, 2023

Upvote!
I am also looking for a good way to get access to the rate limit info that is included in the response headers.

@Gtofig
Copy link

Gtofig commented Mar 4, 2024

+1. Is there any workaround for this while the issue is still unresolved?

@clamey
Copy link

clamey commented Apr 4, 2024 •
edited
Loading

Might be able to do something like this:

      OpenAI(
          "your-open-ai-api-key",
          httpClientConfig = {
            install(
                createClientPlugin("RateLimitPlugin") {
                  onResponse { response ->
                    response.headers["x-ratelimit-remaining-tokens"]?.let { remainingTokens ->
                      response.headers["x-ratelimit-reset-tokens"]?.let { resetTime ->
                        // Do something with the tokens and reset time
                        service.storeTokensAndResetTime(remainingTokens, resetTime)
                      }
                    }
                  }
                },
            )
          },
      )

And maybe add an extension function to OpenAI that wraps the actual call and checks the rate limits:

  suspend fun OpenAI.chatCompletionWithRateLimitCheck(chatCompletionRequest: ChatCompletionRequest): ChatCompletion {
    val (remainingTokens, resetTime) = service.getTokensAndResetTime()
    val requestTokenCount = getMessagesTokenCount(chatCompletionRequest.messages, modelId)
    if (requestTokenCount > remainingTokens) {
      delay(Duration.parse(resetTime).toMillis())
    }
    return openAIClient.chatCompletion(chatCompletionRequest)
  }

  suspend fun getMessagesTokenCount(messages: List<ChatMessage>, modelId: String) = messages
      .map { Tokenizer.of(model = modelId).encode(it.content ?: "") }
      .flatten()
      .sumOf { it }

Just ideas - haven't tested any of this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

update: headers info with functions (Chat + Audio + Completion) Krobys/openai-kotlin
5 participants
@clamey @aallam @Gtofig @palhoye @Krobys