createChatCompletion seems to ignore the abort signal

[MarinaMurashev]

Describe the bug

Sending an 'abort' signal to the createChatCompletion does not raise an error nor stop the completion.

It makes me believe that this discussion on the openai community is true https://community.openai.com/t/cancelling-openai-apis-request/99754, but I would like to verify it isn't a bug in this library.

To Reproduce

Here's my code

import { Configuration, OpenAIApi } from "openai";

const configuration = new Configuration({
  apiKey: process.env.OPENAI_API_KEY,
});
const openai = new OpenAIApi(configuration);

// instantiate the abortController
const abortController = new AbortController();
const abortSignal = abortController.signal;

// send the request, passing in the abortController's signal
const response = await openai.createChatCompletion(
  {
    model: "gpt-3.5-turbo",
    messages: [
      {
        role: "user",
        content: "write a 1000-word essay about George Washington",
      },
    ],
    stream: true,
  },
  {
    // This is an Axios request config object
    responseType: "stream",
    signal: abortSignal,
  }
);

// abort the request in 1 second
const abort = () => {
  abortController.abort();
};
setTimeout(abort, 1000);

// read the response data
const dataStream = response.data as unknown as AsyncIterable<Buffer>;
for await (const chunk of dataStream) {
  if (abortSignal.aborted) {
    console.log("aborted");
  }

  const lines = chunk
    .toString("utf8")
    .split("\n")
    .filter((line) => line.trim().startsWith("data: "));

  for (const line of lines) {
    const message = line.replace("data: ", "");
    if (message === "[DONE]") {
      console.log("done");
    }

    const json = JSON.parse(message);
    const token = json.choices[0]?.delta?.content;
    if (token != null) {
      console.log(`token: ${token}`);
    }
  }
}

Expectation: I should see output like this, and then an error should be raised:

token: George
token:  Washington
token:  is
token:  perhaps
token:  the
token:  most
token:  significant
token:  figure
token:  in
token:  American
token:  history
token: .
aborted

Actual: I see output like this that never stops:

token: George
token:  Washington
token:  is
token:  perhaps
token:  the
token:  most
token:  significant
token:  figure
token:  in
token:  American
token:  history
token: .
aborted
token:  He
aborted
token:  was
aborted
token:  a
aborted
token:  man
aborted
token:  who
aborted
token:  rose
aborted
token:  from
aborted
token:  humble
aborted
token:  beginnings
aborted
token:  to
aborted
token:  become
aborted
token:  the
aborted
token:  first
aborted
token:  President
aborted
token:  of
aborted
token:  the
aborted
token:  United
aborted
token:  States
aborted
token:  of
aborted
token:  America
aborted
token: ,
aborted
token:  and
aborted
token:  he
aborted
token:  did
aborted
token:  so
aborted
token:  with
aborted
token:  a
aborted
token:  level
... (and so on)

Code snippets

No response

OS

macOS

Node version

v19.8.1

Library version

openai v3.2.1

[numeralz]

I have tried both cancelToken and signal: abortController neither stop the completion.
Especially frustrating when the bot goes into a loop repeating the same line.

By the way, I tried using frequency and presence penalty, but I think those penalty scores cause the bot to compensate with incorrect substitute tokens in things like long JSON responses.

[wtesler]

I am having the same issue. Perhaps it related to the version of axios that they are using (0.X.X) which may not support cancellation of a streaming response.

It may be the case that cancellation of a streaming response was added in 1.X of axios here: axios/axios#4772

[justinmahar]

I ran into the same issue trying both, seems to be a bug.

Your mileage may vary, but as a workaround I was able to successfully stop completions by calling destroy() on the stream
via response.data.destroy().

[Geczy]

same issue here, any solutions?

[numeralz]

I had some luck using

  const data = {
    messages,
    model,
    stream      : true,
    temperature : 0,
  };
  
  const abortController = new AbortController ();
  
  fetch (
    "https://api.openai.com/v1/chat/completions",
    {
      method  : "POST",
      headers : {
        "Authorization" : `Bearer ${ openaiKey }`,
        "Content-Type"  : "application/json",
      },
      body   : JSON.stringify ( data ),
      signal : abortController.signal,
    }
  )

[lxmfly123]

It appears that destroying the stream is the only necessary action.

axios/axios#499

[rattrayalex]

Thanks for the report! Our upcoming version v4.0.0 should properly support the abort signal; please give it a try and share your feedback in the thread!

[lxmfly123]

openai sdk v3 is using axios v0.26.1, which is unable to stop stream under node environment

[rattrayalex]

Please give openai sdk v4 a try:

npm install [email protected]

[and-zverev]

Please give openai sdk v4 a try:

npm install [email protected]

Hey @rattrayalex

I use [email protected] in my NestJS app like this:

const abortController = new AbortController();
const stream = await openai.chat.completions.create(
  {
    model: 'gpt-4',
    messages: [{ role: 'user', content: 'test prompt' }],
    stream: true
  },
  { signal: abortController.signal }
);
setTimeout(() => {
  // stream.controller.abort();
  abortController.abort();
}, 1000);

But when I try to make a production build of the project, I get Uncaught ReferenceError: AbortController is not defined exception raised in node_modules/openai/_shims/agent.node.js

// node_modules/openai/_shims/agent.node.js
if (typeof AbortController === 'undefined') {
  AbortController = abort_controller_1.AbortController;
}

Could you help me figure this out?

[rattrayalex]

@and-zverev can you open a new issue with this information, and also include your package.json, tsconfig, and similar files?

[and-zverev]

I just double-checked everything and found that I worked on [email protected]
it works great on [email protected]
sorry to bother you, my fault