Allow to handle stream cancellation on server

[jvandenaardweg]

Now we have stream cancellation working properly (#90 (comment) ) I think we need a way to detect and handle the cancellation server side.

In the current situation when a user cancels a stream, the server errors and there's no way to catch the cancellation and handle what should happen on the server. But I want to know how many tokens are generated when the user cancels the stream, so I can then store that information in my external database.

Currently we have onStart, onCompletion and onToken callback methods on OpenAIStream() / AIStream(). Ideally a new callback method would be great, something like onStop (to follow the stop() method naming in the React Hooks) or onCancel, which passes through the completion text at that point and allows to calculate the tokens and write that to a external database.

Or a different way to just catch the abort error in our (Edge) Functions.

From my understanding, I guess we need to postpone the abort until whatever method is doing something async is done, otherwise the serverless function might not exist anymore.

Is this even possible?

[hassanbazzi]

Having the same issue. This is very important.

[gartia]

This would be nice. to have an OnStop. I am finding it annoying to deal with function calls, when i want to abort after a function call. But i getting errors from using abort controllers via server methods.

[lassegit]

Is it really necessary to have all these callbacks? Having onStop(completion, error) call should be enough. My problem is that I don't have a callback when the abort signal is called. I need a callback to be able to track token usage.

[GorvGoyl]

What are some current workarounds till the team fixes the issue?

[GorvGoyl]

related issue vercel/next.js#48381

[peterdresslar]

Hello @GorvGoyl, are you calling the OpenAI API using the OpenAI Node library (as documented in the examples from this SDK) or using your own custom logic to access the API?

[GorvGoyl]

Hello @GorvGoyl, are you calling the OpenAI API using the OpenAI Node library (as documented in the examples from this SDK) or using your own custom logic to access the API?

I'm using vercel edge function like this:

 const completion = await openai.chat.completions.create({
            model: 'gpt-3.5-turbo',
            stream: true,
            messages: [{ role: 'user', content: question }],
        });

        const stream = OpenAIStream(completion, {...});
        const streamResponse = new StreamingTextResponse(stream);
            return streamResponse;

and then in the frontend (chrome extension) I'm using ReadableStream and TextDecoder to parse (can't use useChat or useCompletion in extension)

[jvandenaardweg]

related issue vercel/next.js#48381

Yes, that's the same issue. Once aborted, you get a uncaught exception, which you can't catch.

What are some current workarounds till the team fixes the issue?

I mentioned 2 work arounds here: #90 (comment)

1. When a user initiates a cancel from your UI, send the AI generated text to an API that tracks the token/character/word usage of that text. Probable downside is when a user navigates away from your site (or closes the browser), then the API call will not be fired.
2. Track each generated token using onToken on the server and after a certain timeout just assume its cancelled. Also track the completion, so you can cancel them out. But for this you need to keep a record in a Redis store or something. Send each token to that store (that are a lot of requests depending on the size of text being generated). Because once the stream is done, the Edge Function is terminated and your timeout code will not run. Then I guess you need to poll Redis each minute or so and pull that usage info into your main database.

I currently have option 1 in place in my app. Not optimal, but in terms of dev time the easiest I think.

To be honest, I don't think this catching of cancellation on the server - and then do something async with it - will ever work with serverless functions. Please prove me wrong! The only real permanent solution is using a normal "always on" server, like a NodeJS Express server or something else that's always running. Because when it's always running, you can do async things with a cancellation.

[yvngjordi]

Still having the same problem. Guess Vercel SDK is just limited

[winddpan]

Hope Vercel can solve it.

[lgrammel]

The AI SDK functions support a abortSignal option that you can use to forward the abort.

[verrannt]

Could you elaborate a bit how to achieve the desired functionality using abortSignal? I am facing the same problem as OP and it is not at all clear to me how I would go about implementing it with abortSignal.

Or anyone else who has successfully implemented a workaround to achieve something like an onStop callback for manually stopped responses?

[xl0]

@lgrammel It Would be nice if abortSignal could trigger onFinish and pass as much data as possible to it. This would make it easier to save aborted messages to the database.

[lgrammel]

@xl0 this will be problematic because of typesafety and cause a lot of optionality checks. I would prefer something likeonAbort - would that work for you?

[xl0]

@lgrammel onAbort would work perfectly fine for me.