[.Net][Feature Request]: Ollama API support

[iddelacruz]

Is your feature request related to a problem? Please describe.

No response

Describe the solution you'd like

Add a AutoGen.Ollama package for Ollama API support

Tasks

• Add AutoGen.Ollama package
• Add more test cases and comment
  • test for multi-moda model
  • test for ollama message connector
  • test for embedding api
  • end2end agent tests
• Add document/sample for chat

[mikelor]

Oops! I just noticed this after submitting Pull Request #2512

I basically added support for Ollama via it's OpenAI compatible API.
It's a short-term fix IMO, the current architecture is heavily Azure OpenAI specific, but not letting perfect be the enemy of good, created an OllamaAgent and Config.

using AutoGen.Core;
using AutoGen.Ollama;

var config = new OllamaConfig("localhost", 11434);

// You can specify any model Ollama supports.
// See list here: https://ollama.com/library
// Just make sure you "pull" the model using "ollama pull" first.
var assistantAgent = new OllamaAgent("asssistant", config: config, "llama3")
    .RegisterPrintMessage();

// set human input mode to ALWAYS so that user always provide input
var userProxyAgent = new UserProxyAgent(
    name: "user",
    humanInputMode: HumanInputMode.ALWAYS)
    .RegisterPrintMessage();

// start the conversation
await userProxyAgent.InitiateChatAsync(
    receiver: assistantAgent,
    message: "Why is the sky blue?",
    maxRound: 10);

Console.WriteLine("Thanks for using Ollama. https://ollama.com/blog/");
'''

[LittleLittleCloud]

@mikelor Thanks for the PR/contributing, it's very important to us.

The ollama backend is not 100% identical with openai chat completion scheme, so we still want a 100% capacity ollama client in AutoGen.Ollama package (like what AutoGen.Mistral does with MistralClient)

Maybe you can consider modifying your PR to add support for consuming third-party openai backend in OpenAIClientAgent by adding a constructor like below? It's up to you though

public OpenAIChatAgent(
    Uri endpoint, // openai chat backend
    string name,
    string modelName,
    string systemMessage = "You are a helpful AI assistant",
    float temperature = 0.7f,
    int maxTokens = 1024,
    int? seed = null,
    ChatCompletionsResponseFormat? responseFormat = null,
    IEnumerable<FunctionDefinition>? functions = null)
{
    this.openAIClient = openAIClient;
    this.modelName = modelName;
    this.Name = name;
    _temperature = temperature;
    _maxTokens = maxTokens;
    _functions = functions;
    _systemMessage = systemMessage;
    _responseFormat = responseFormat;
    _seed = seed;
}

[mikelor]

@LittleLittleCloud, I think modifying the PR to support an OpenAIClientAgent is the best route for Ollama.

The Mistral client is used for consuming the Mistral model. Ollama is a local LLM server capable of serving multiple models Llama, Phi, and others (including Mistral).

Therefore, leveraging the OpenAIClientAgent seems to make more sense. That's what I was trying to do with the Pull Request, but looking at your suggestion, I'll revisit the current implementation.

Thoughts?

[LittleLittleCloud]

@mikelor the OpenAI chat completion api support in ollama is in experimental and not 100% compatible, please check the following link for more information

https://github.com/ollama/ollama/blob/main/docs/openai.md#openai-compatibility

And I feel like it would be better for the AutoGen.Ollama to have a 100% support for ollama backend, which can’t be achieved by leveraging ‘OpenAIClientAgent’ and essentially means we need an ollama client.

what you think

[mikelor]

I would agree, I'll take another look at your comments above, maybe I misunderstood, but easy to do given the medium of interaction.

I tried to follow the pattern set forth in the LMStudio implementation because Ollama is closer to that (supports hosting multiple models).

Also, given the experimental nature of OpenAI Support in Ollama, it was more important to get something working, and then iterate as support grows.

I'm going to take some time to review autogen's agent/plugin architecture, before submitting any further changes. Given the time available, it will take a week or so.

Maybe ask some questions on the discord as well, in the #dotnet channel I'm guessing.

[LittleLittleCloud]

Sounds good, Your PR brings a very good point, which is how to consume a third-party openai endpoint in OpenAIClientAgent. And I think it would be a very useful feature for OpenAIClientAgent because nearly all model provider support openai chat scheme in some way (LMStudio, Mistral, Ollama, vllm,,,,,). Like what you said, it was more important to get something working, supporting third-party openai endpoint in OpenAIClientAgent would make it work with not just Ollama, but nearly all model providers.

Update on 2024/05/10

For those who wants to connect to an openai compatible API using OpenAIClientAgent, please refer to https://microsoft.github.io/autogen-for-net/articles/OpenAIChatAgent-connect-to-third-party-api.html on how to do it