Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

完善api.py,增加LLM对话和知识库对话的流式输出功能 #659

Merged
merged 6 commits into from
Jul 25, 2023
Merged

Conversation

margox
Copy link
Contributor

@margox margox commented Jun 18, 2023

/chat/local_doc_qa/local_doc_chat这两个接口增加了stream参数,默认为false,传true则启用流式输出(输出的是纯文本内容,所以数据结构和非流式输出的时候完全不同)

这个入参方式和流式数据处理,跟OpenAI的chat是一致的,对于之前使用OpenAI接口的项目来说,应该是比较容易接入的。

下面是一个在浏览器端调用API进行流式对话的代码示例:

const TIME_OUT_MS = 30000

export const requestStreamingChat = async (params) => {
  const { question, history, onError, onMessage, onAbortController } = params

  const controller = new AbortController()
  const reqTimeoutId = setTimeout(() => controller.abort(), TIME_OUT_MS)

  try {
    const res = await fetch("http://127.0.0.1:7861/chat/", {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
      },
      signal: controller.signal,
      body: JSON.stringify({
        question,
        history,
        stream: true, // 这里开启了stream
      }),
    })

    clearTimeout(reqTimeoutId)

    let responseText = ''

    const finish = () => {
      onMessage(responseText, true)
      controller.abort()
    }

    if (res.ok) {
      const reader = res.body?.getReader()
      const decoder = new TextDecoder()

      onAbortController?.(controller)

      while (true) {
        const resTimeoutId = setTimeout(() => finish(), TIME_OUT_MS)
        const content = await reader?.read()
        clearTimeout(resTimeoutId)
        const text = decoder.decode(content?.value)
        responseText += text

        const done = !content || content.done
        onMessage(responseText, false)

        if (done) {
          break
        }
      }

      finish()
    } else {
      onError(new Error('Stream Error'), res.status)
    }
  } catch (err) {
    console.log(err)
    console.error('NetWork Error', err)
    onError(err as Error, 'Network Error')
  }
}

@imClumsyPanda imClumsyPanda changed the base branch from master to dev June 18, 2023 13:54
@liunux4odoo liunux4odoo merged commit 18d453c into chatchat-space:dev Jul 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants