Custom inline completion providers

[zerocorebeta]

Summary: Custom inline completion providers for local models or other platforms

--

After going through: https://zed.dev/docs/completions

Zed currently supports completions via external LLM APIs like GitHub Copilot and Supermaven, but this is restrictive. Many users, for privacy or performance reasons, might prefer alternatives like Gemini Flash or local models via Ollama.

There are several advanced LLMs that support the Fill-in-the-Middle (FIM) objective, such as CodeGemma. Additionally, platforms like Continue.dev allow code completion via local models, with Startcoder via Ollama as the default.

Expanding LLM support to include more flexible, local, or privacy-focused options would greatly enhance Zed's appeal and utility for a wider range of developers.

[ggerganov]

We have recently extended the llama.cpp server with a specialized /infill endpoint that enables FIM requests with large contexts to run efficiently in local environments. A simple example of using this endpoint with Qwen2.5-Coder in Neovim is demonstrated here: ggml-org/llama.cpp#9787

I believe it could be an interesting option to explore in the scope of this issue. Feel free to ping me if you have any questions.

Edit: sample client-side plugins using the llama.cpp server:

• Vim/Neovim: https://github.com/ggml-org/llama.vim
• VS Code: https://github.com/ggml-org/llama.vscode

These can be used as reference implementations for Zed support.

[20manas]

I think Zed AI should also provide its own code completion functionality.

[bersace]

Does Zed allows to use Codestral cloud for FIM ?

[josharian]

https://arstechnica.com/ai/2025/01/nvidias-first-desktop-pc-can-run-local-ai-models-for-3000/ is relevant here. FIM models aren't all that big; one of these could probably handle an entire office's worth of requests.

I'd really like to be able to use a llama.cpp model for FIM.

[aretrace]

Configurable FIM support should be a priority. While the improvements to the Assistant panel over time have been great, the lack of customizable AI autocomplete in Zed is becoming a significant drawback compared to the code assistance experiences offered by others, such as IntelliSense.

[mbitsnbites]

I'm currently using a locally running llama.cpp server with Qwen2.5-Coder-7B and managed to get zed to use it for the assistant and inline assistant (it works really well), but I'm very much missing the inline completion support.

As a side note, I found the configuration a bit confusing so I'm sharing it here (here "api_url" points to my local llama.cpp server instance):

• Configure settings.json as follows:

  "language_models": {
    "openai": {
      "version": "1",
      "api_url": "http://localhost:8081",
      "available_models": [
             {
               "name": "qwen2.5-coder-7b",
               "display_name": "Qwen2.5-Coder-7B",
               "max_tokens": 128000
             }
      ]
    }
  },
  "assistant": {
    "version": "2",
    "default_model": {
      "provider": "openai",
      "model": "qwen2.5-coder-7b"
    }
  },

• Configure the OpenAI field in the assistant configuration panel by adding a dummy API key.

Edit: I could not find any configuration options in settings.json relating to inline completion. My zed currently appears to be using GitHub Copilot for inline completion (I activated that before setting up my local llama.cpp connection, and I can't find a way to deactivate it), but llama.cpp is used for the assistant and inline assistant.

[kylelee]

IMO, the feature (ollama as an inline_completion_provider) is very important to every developer work in LAN.