Workaround for litellm proxy -> ollama issue #377
Conversation
up an OpenAI API compatible proxy to ollama. In this case, the last chunk was set to None, which throws an exception when we try appending it to the content string.
|
I've researched the issue more now, and it's the very last chunk, containing the finish_reason field, that in the case of at least LiteLLM:s implementation of the OpenAI API, also sets the 'delta' field to {content: null, role: null} rather than just not setting the delta field at all in this case. I think it's a good idea for this project to be resilient to that type of slight deviation from how the real OpenAI endpoints might respond, but I also think it should be fixed upstream in the litellm repo. I've already made some pull requests for a couple of other issues there when using litellm with ollama. |
|
Sorry for not getting this into 0.13 - this will need some tweaks now that I've upgraded the OpenAI Python library. |
|
Hey @clevcode i'm the maintainer of litellm Tracking this issue on our end as well - BerriAI/litellm#2010 Thanks for raising it. |
When streaming responses from a model served by ollama, through a litellm proxy that provides an OpenAI compatible API, the last chunk (i.e. choice["delta"]["content"]) of the streaming response will be set to None
This causes an exception to be thrown during the following, since None can't be appended to a string:
content += choice["delta"]["content"]
To avoid this, I simply added a check so that choice["delta"]["content"] is only appended to content if it's not None.