Flask streaming OpenAI

[sorrentinopasquale]

Hi all, I searched a lot online but still no valid final solution.
I need to stream each token but I don't find the right way yet.
So.. all works very fine, and I am able since I override on_llm_new_token to see also in console the streaming.
How can I now send it to the client? I know I can not use yield in the callback, is there any easier way instead of managing async calls?

Thanks a lot for the support

`
class CustomStreamingCallbackHandler(StreamingStdOutCallbackHandler):
def on_llm_new_token(self, token: str, **kwargs: Any) -> None:
sys.stdout.write(token) # Modify the behavior here
sys.stdout.flush()

@flow_call_streaming_api_blueprint.route('/streaming/call')
def stream_chat_gpt():
"""
This streams the response from ChatGPT
"""
prompt = request.args.get('prompt')
projectId = request.args.get('projectId')
sender = request.args.get('ip', default='00000')

response = Response(stream_with_context(chat_gpt_helper(prompt, projectId, sender)),
                    mimetype='text/event-stream')
response.headers['Cache-Control'] = 'no-cache'

return response

def chat_gpt_helper(prompt, projectId, sender):
try:
custom_callback_handler = CustomStreamingCallbackHandler() # Instantiate the callback handler

    llm_chain = LLMChain(
        llm=OpenAI(model_name="gpt-4", temperature=0.3, streaming=True, callbacks=[custom_callback_handler]),
        prompt=prompt_template,
        verbose=True,
        memory=memory,
    )

    result = llm_chain.run(prompt)
    print(result)

except Exception as e:
    print(e)
    return str(e)

`

[dosubot[bot]]

🤖

Hello,

Based on your code and the context provided, it seems you're trying to stream tokens from the OpenAI model to the client in Flask. Here's a way to do it:

First, you need to set up a custom callback handler that will handle each new token. You've already done this with your CustomStreamingCallbackHandler class.

Next, you need to set up a Flask route that will generate a streaming response. Here's an example of how you can do this:

from flask import Response
from langchain.callbacks.manager import CallbackManager
from langchain.llms.openai import OpenAI
from your_module import CustomStreamingCallbackHandler

@app.route('/stream')
def stream():
    async def generate():
        callback_handler = CustomStreamingCallbackHandler()
        callback_manager = CallbackManager([callback_handler])
        llm = OpenAI(streaming=True, callback_manager=callback_manager, temperature=0)
        message = "Hello, world!"
        async for chunk in llm(message):
            yield chunk
    return Response(generate(), mimetype='text/plain')

In this example, a new OpenAI instance is created with the streaming parameter set to True and the CallbackManager passed in the callback_manager parameter. The generate function yields each token as it is received from the OpenAI API, and this function is passed to the Response object to create a streaming response.

Please note that this is a simplified example and may need to be adjusted based on your specific use case. For example, you may need to handle errors and retries, and you may want to customize the parameters passed to the OpenAI constructor.

For more information, you can refer to the following sources:

• libs/langchain/langchain/llms/openai.py
• docs/snippets/modules/model_io/models/llms/how_to/streaming_llm.mdx
• libs/langchain/langchain/callbacks/streaming_aiter.py
• libs/langchain/tests/integration_tests/chat_models/test_openai.py
• libs/langchain/tests/integration_tests/llms/test_openai.py

I hope this helps! If you have any other questions, feel free to ask.

Sources

• Support for streaming when using LLMchain?
• libs/langchain/langchain/llms/openai.py
• docs/snippets/modules/model_io/models/llms/how_to/streaming_llm.mdx
• libs/langchain/langchain/callbacks/streaming_aiter.py
• libs/langchain/tests/integration_tests/chat_models/test_openai.py
• libs/langchain/tests/integration_tests/llms/test_openai.py
• libs/langchain/langchain/callbacks/streaming_stdout_final_only.py
• libs/langchain/langchain/callbacks/streaming_stdout.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.