community[minor]: Add UpstashRatelimitHandler #21885

CahidArda · 2024-05-19T18:38:56Z

Adding UpstashRatelimitHandler callback for rate limiting based on number of chain invocations or LLM token usage.

For more details, see upstash/ratelimit-py repository or the notebook guide included in this PR.

Twitter handle: @CahidArda

vercel · 2024-05-19T18:39:01Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchain	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 7, 2024 9:02pm

eyurtsev

looks good overall, some minor comments only

libs/community/langchain_community/callbacks/upstash_ratelimit_callback.py

eyurtsev · 2024-05-20T18:20:35Z

libs/community/langchain_community/callbacks/upstash_ratelimit_callback.py

+        Example:
+            .. code-block:: python
+
+            from upstash_redis import Redis


missing indentation code block example

eyurtsev · 2024-05-20T18:23:01Z

libs/community/langchain_community/callbacks/__init__.py

@@ -72,6 +72,10 @@
    from langchain_community.callbacks.trubrics_callback import (
        TrubricsCallbackHandler,
    )
+    from langchain_community.callbacks.upstash_ratelimit_callback import (
+        UpstashRatelimitError,  # noqa: F401


Could you remove the F401 please to match the other callbacks?

libs/community/pyproject.toml

eyurtsev · 2024-05-20T21:09:59Z

After thinking a bit more about this -- I am not sure that this is a good design for rate limiting.

Why is it implemented via a callback handler? It would ideally just be a part in the chain that can wait until it can issue a request?

CahidArda · 2024-05-21T09:01:41Z

After thinking a bit more about this -- I am not sure that this is a good design for rate limiting.

Why is it implemented via a callback handler? It would ideally just be a part in the chain that can wait until it can issue a request?

I wanted to use callbacks because I felt like it would make adding request or token based ratelimiting very easy.

I guess something like this would work for request based rate limiting:

# request based
request_limiter = UpstashRatelimit("ip")
other_step = RunnableLambda(str)

chain = request_limiter | other_step
chain.invoke()

But I think token based would be more complex. We would need a step before LLM starts to stop the chain and another step after the LLM to count the tokens. Or somehow wrap the model step to do both but I don't know if this is possible in LangChain

# token based
other_step = RunnableLambda(str)
model = ChatOpenAI()
model_with_ratelimit = UpstashRatelimit("ip", model=model)

chain = other_step | model_with_ratelimit
chain.invoke()

eyurtsev · 2024-05-22T18:36:08Z

libs/community/poetry.lock

@@ -1,4 +1,4 @@
-# This file is automatically @generated by Poetry 1.7.1 and should not be changed by hand.
+# This file is automatically @generated by Poetry 1.8.2 and should not be changed by hand.


could you undo the changes in the lock file?

eyurtsev · 2024-05-22T18:40:14Z

We generally don't want to assume that callbacks must be blocking for execution.

What use case is this callback handler helping to solve given that it's raising an exception?

Is the goal to apply different (lower) rate limits on a given deployment then the ones specified by the model provider?

CahidArda · 2024-05-22T20:30:40Z

We generally don't want to assume that callbacks must be blocking for execution.

What use case is this callback handler helping to solve given that it's raising an exception?

Is the goal to apply different (lower) rate limits on a given deployment then the ones specified by the model provider?

Yes, with the callback, it's becomes possible to allow n number of requests from an ip address or some user per minute/hour/day. It's also possible to rate limit based on the number of tokens.

eyurtsev · 2024-05-23T15:39:38Z

I suspect a better design would be to create a chat model wrapper, potentially a bit more work for the user, but won't have any unexpected issues associated with the callback not being blocking

@CahidArda Anyway, let me know if you'd still like to merge -- if so could you remove the changes from the lock file? (i assume they're unnecessary for this PR?)

This reverts commit 687b780.

CahidArda · 2024-05-25T22:32:52Z

Hi @eyurtsev,

I think we can go ahead with callback if it's okay.

As for the lockfile, I have tried to remove it but when I remove it linter gets an error saying that the lock file is not compatible with the toml file. If I remove the changes in the toml file, tests get an error saying that upstash_ratelimit was not found. So I added upstash_ratelimit and bumped upstash_redis version while I am at it.

CahidArda · 2024-05-29T07:26:21Z

Hi @eyurtsev,

Have you had the chance to review the changes?

CahidArda · 2024-06-01T20:20:46Z

Hi again @eyurtsev,

JavaScript version of this PR was merged recently. Have you had a chance to review the latest changes in this PR? 😄

eyurtsev · 2024-06-05T15:39:48Z

@CahidArda apologies was on vacation until yesterday! merging

libs/community/tests/unit_tests/callbacks/test_upstash_ratelimit_callback.py

libs/community/langchain_community/callbacks/upstash_ratelimit_callback.py

eyurtsev · 2024-06-05T15:55:26Z

@CahidArda could you address the side-effects for the optional imports and we can merge then?

vercel · 2024-06-06T05:56:47Z

Deployment failed with the following error:

The provided GitHub repository does not contain the requested branch or commit reference. Please ensure the repository is not empty.

CahidArda · 2024-06-06T06:30:46Z

Hope you had a great holiday! 🌴

I fixed the side effects.

eyurtsev · 2024-06-07T20:45:53Z

Taking over to resolve merge conflicts

@CahidArda

Adding `UpstashRatelimitHandler` callback for rate limiting based on number of chain invocations or LLM token usage. For more details, see [upstash/ratelimit-py repository](https://github.com/upstash/ratelimit-py) or the notebook guide included in this PR. Twitter handle: @CahidArda --------- Co-authored-by: Eugene Yurtsev <[email protected]>

add UpstashRatelimitHandler

df71c4b

dosubot bot added the size:XL This PR changes 500-999 lines, ignoring generated files. label May 19, 2024

dosubot bot added the 🤖:improvement Medium size change to existing code to handle new use-cases label May 19, 2024

vercel bot deployed to Preview May 19, 2024 18:50 View deployment

CahidArda force-pushed the upstash-callback branch 2 times, most recently from ab4dd42 to 13fd6c7 Compare May 19, 2024 19:13

add upstash ratelimit callback guide

821f867

CahidArda force-pushed the upstash-callback branch from 13fd6c7 to 821f867 Compare May 19, 2024 19:22

vercel bot deployed to Preview May 19, 2024 19:36 View deployment

Fix typo

11d814d

vercel bot deployed to Preview May 20, 2024 06:48 View deployment

CahidArda mentioned this pull request May 20, 2024

community[minor]: Add UpstashRatelimitHandler langchain-ai/langchainjs#5474

Merged

add upstash-ratelimit dependency

b66a339

CahidArda force-pushed the upstash-callback branch from 3451595 to b66a339 Compare May 20, 2024 07:38

vercel bot deployed to Preview May 20, 2024 07:50 View deployment

eyurtsev reviewed May 20, 2024

View reviewed changes

libs/community/pyproject.toml Outdated Show resolved Hide resolved

baskaryan assigned eyurtsev May 20, 2024

fmt

5d1f62d

vercel bot deployed to Preview May 22, 2024 16:41 View deployment

eyurtsev reviewed May 22, 2024

View reviewed changes

eyurtsev added the waiting-on-author PR Status: Confirmation from author is required label May 23, 2024

eyurtsev approved these changes May 23, 2024

View reviewed changes

dosubot bot added the lgtm PR looks good. Use to confirm that a PR is ready for merging. label May 23, 2024

vercel bot deployed to Preview May 25, 2024 17:04 View deployment

CahidArda added 2 commits May 26, 2024 01:04

revert changes in poetry.lock and pyproject.toml files

687b780

Revert "revert changes in poetry.lock and pyproject.toml files"

5547fd5

This reverts commit 687b780.

vercel bot deployed to Preview May 25, 2024 22:28 View deployment

Merge branch 'master' into upstash-callback

8cdb2f3

eyurtsev changed the title ~~community: Add UpstashRatelimitHandler~~ community[minor]: Add UpstashRatelimitHandler Jun 5, 2024

eyurtsev enabled auto-merge (squash) June 5, 2024 15:39

vercel bot deployed to Preview June 5, 2024 15:47 View deployment

relock

c6a5c27

eyurtsev disabled auto-merge June 5, 2024 15:53

eyurtsev reviewed Jun 5, 2024

View reviewed changes

libs/community/tests/unit_tests/callbacks/test_upstash_ratelimit_callback.py Outdated Show resolved Hide resolved

libs/community/langchain_community/callbacks/upstash_ratelimit_callback.py Outdated Show resolved Hide resolved

vercel bot deployed to Preview June 5, 2024 16:01 View deployment

rm optional import side-effects

23b901a

rm named exception

dbc6ca4

vercel bot deployed to Preview June 6, 2024 06:08 View deployment

eyurtsev added 2 commits June 7, 2024 16:47

Merge branch 'master' into upstash-callback

384d5dc

x

2ad9cb3

eyurtsev enabled auto-merge (squash) June 7, 2024 20:54

vercel bot deployed to Preview June 7, 2024 21:02 View deployment

eyurtsev merged commit 6c07eb0 into langchain-ai:master Jun 7, 2024
44 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

community[minor]: Add UpstashRatelimitHandler #21885

community[minor]: Add UpstashRatelimitHandler #21885

CahidArda commented May 19, 2024

vercel bot commented May 19, 2024 •

edited

Loading

eyurtsev left a comment

eyurtsev May 20, 2024

eyurtsev May 20, 2024

eyurtsev commented May 20, 2024

CahidArda commented May 21, 2024

eyurtsev May 22, 2024

eyurtsev commented May 22, 2024

CahidArda commented May 22, 2024

eyurtsev commented May 23, 2024 •

edited

Loading

CahidArda commented May 25, 2024

CahidArda commented May 29, 2024

CahidArda commented Jun 1, 2024

eyurtsev commented Jun 5, 2024

eyurtsev commented Jun 5, 2024

vercel bot commented Jun 6, 2024

CahidArda commented Jun 6, 2024

eyurtsev commented Jun 7, 2024

		@@ -1,4 +1,4 @@
		# This file is automatically @generated by Poetry 1.7.1 and should not be changed by hand.
		# This file is automatically @generated by Poetry 1.8.2 and should not be changed by hand.

community[minor]: Add UpstashRatelimitHandler #21885

community[minor]: Add UpstashRatelimitHandler #21885

Conversation

CahidArda commented May 19, 2024

vercel bot commented May 19, 2024 • edited Loading

eyurtsev left a comment

Choose a reason for hiding this comment

eyurtsev May 20, 2024

Choose a reason for hiding this comment

eyurtsev May 20, 2024

Choose a reason for hiding this comment

eyurtsev commented May 20, 2024

CahidArda commented May 21, 2024

eyurtsev May 22, 2024

Choose a reason for hiding this comment

eyurtsev commented May 22, 2024

CahidArda commented May 22, 2024

eyurtsev commented May 23, 2024 • edited Loading

CahidArda commented May 25, 2024

CahidArda commented May 29, 2024

CahidArda commented Jun 1, 2024

eyurtsev commented Jun 5, 2024

eyurtsev commented Jun 5, 2024

vercel bot commented Jun 6, 2024

CahidArda commented Jun 6, 2024

eyurtsev commented Jun 7, 2024

vercel bot commented May 19, 2024 •

edited

Loading

eyurtsev commented May 23, 2024 •

edited

Loading