epic: Refactor Inference Engines

[louis-menlo]

Motivation

Currently, integrating new inference providers involves significant code duplication, including duplicating SSE helper functions, handling engine settings file I/O operations, and managing event handling for loading, unloading models, and inference processes.

Specs

• Implementing a structured inheritance model to eliminate redundant code by inheriting from a base inference engine class and dynamically registering engine-specific settings.
• Encouraging individual inference engines to manage their models, rather than having the application handle all models under a single directory (/models). This approach enhances model and engine maintenance and facilitates migration.
• Shifting from hardcoded API key inputs to registering proper settings from the extension level, improving flexibility and security.
• Addressing the scalability issues caused by numerous if-else statements and engine name comparisons in the frontend. This refactor aims to make it easier to support new engines and enhance scalability.

Tasklist

• Identify and analyze areas of code duplication related to inference engine integration. (feat: Nitro-Tensorrt-LLM Extension #2280 fix: refactor inference engines to extends AIEngine #2347)
• Design and implement a structured inheritance model for inference engines to promote code reuse. (fix: refactor inference engines to extends AIEngine #2347)
• Refactor existing inference engine implementations to inherit from the base class and remove duplicated code. (fix: refactor inference engines to extends AIEngine #2347)
• Develop a system for individual inference engines to manage their models independently. (feat: Nitro-Tensorrt-LLM Extension #2280)
• Inference Engine Manager now grants frontend access to engines for controlling logic flow (refactor: inference engine provider and accessibility #2470)
• Refactor assistant's tools to register the tool instead of intercepting message requests since we don't broad cast completion message anymore (refactor: introduce inference tools #2493)
• Create a mechanism to register engine settings dynamically from the extension level. (feat: Extension should be able to register its own settings #776)
• Provide registerModels. Refactor Inference Extensions to use registerModels function. @namchuai
• Refactor Model Extension to support import, convert models (deprecate huggingface extension) @namchuai
• Improve frontend logic to reduce reliance on if-else statements and engine name comparisons. @namchuai

Out of scope

• Preconfigured models defined inb /jan/engines/mistral/models (KIV? current under /models)

• User can choose the models from the Input Box (KIV?)

• Mistral Extension defines the Right Panel settings that user can change. i.e. Mistral Extension “registers” the Right Panel Settings view (KIV?)

• Add more providers:

• Local:

  • llama.cpp (i.e. Cortex)
  • MLX
  • TensorRT-LLM, Triton Inference Server (feat: Nitro-Tensorrt-LLM Extension #2280)

• Remote:

  • OpenAI
  • Claude,
  • Gemini
  • Mistral (feat: inference-mistral-extension #2072)
  • Groq (feat: Groq Inference Extension #2263)
  • Fireflies
  • Together.ai
  • ...

• Other Libraries

  • Ollama
  • LMStudio
  • ..

Not in Scope

• Customize UI at the extensions level

[louis-menlo]

cc @namchuai @metaspartan

[hiro-v]

• Gemini does not have openai compatible API: https://ai.google.dev/tutorials/rest_quickstart
• Claude: https://docs.anthropic.com/claude/reference/messages_post -> openai compatible, please skip legacy completion api
  • However, Claude supports via Vertex AI (https://docs.anthropic.com/claude/reference/claude-on-vertex-ai) and AWS Bedrock (https://docs.anthropic.com/claude/reference/claude-on-amazon-bedrock) - similar to Azure OpenAI as well. We might want to support it too
• Ollama supported openai compatible api as well: https://ollama.com/blog/openai-compatibility

[louis-menlo]

Since we scoped down the epic and create according out-of-scope sub tasks. Let's close this. cc @Van-QA

[dmatora]

No hope for Gemini support anytime soon?
I'm using Gemini 1.5 Pro via Vertex AI and getting results way above what GPT4 can do, would really love to have proper chat interface