LLM-router

Overview

Access models from OpenAI, Groq, local Ollama, and other providers by setting LLM-router as the base URL in Cursor.

LLM-router is a reverse proxy that routes chat/completions API requests to various OpenAI-compatible backends based on the model's prefix.

Background

Cursor lacks support for local models and the flexibility to switch between multiple LLM providers efficiently. There's clear demand in the community, evidenced by months of unanswered requests for such features on Cursor's GitHub and Discord channels.

The specific challenges with Cursor include:

1. The Override OpenAI Base URL setting requires a URL that is publicly accessible and secured with https, complicating the use of local models.
2. The platform allows configuration for only one LLM provider at a time, which makes it cumbersome to switch between service providers.

LLM-router overcomes these limitations, allowing seamless switching between locally served models like Ollama and external services such as OpenAI and Groq.

llm-router-cursor-preview-web.mp4

Getting Started

1. Launch LLM-router to manage API requests across multiple backends:

OPENAI_API_KEY=<YOUR_OPENAI_KEY> ./llm-router-darwin-arm64

2. Launch ngrok to create a public HTTPS endpoint for your LLM-router:

ngrok http 11411

Configure the Override OpenAI Base URL in Cursor's model settings to point to your ngrok address appended with /v1:

https://xxxx.ngrok-free.app/v1

Define your preferred models:

ollama/phi3
ollama/llama3:70b

Details

Routes chat/completions API requests to any OpenAI-compatible LLM backend based on the model's prefix. Streaming is supported.

LLM-router can be set up to use individual API keys for each backend, or no key for services like local Ollama.

By default, requests to LLM-router are secured with your OPENAI_API_KEY as Cursor already includes this with every request.

Here is an example of how to configure Groq, Ollama, and OpenAI backends in config.json:

{
	"listening_port": 11411,
	"backends": [
		{
			"name": "openai",
			"base_url": "https://api.openai.com",
			"prefix": "openai/",
			"default": true,
			"require_api_key": true
		},
		{
			"name": "ollama",
			"base_url": "http://localhost:11434",
			"prefix": "ollama/"
		},
		{
			"name": "groq",
			"base_url": "https://api.groq.com/openai",
			"prefix": "groq/",
			"require_api_key": true,
			"key_env_var": "GROQ_API_KEY"
		}
	]
}

In this configuration, OpenAI serves as the default backend, allowing you to use model identifiers like openai/gpt-4-turbo or simply gpt-4-turbo. Models on Ollama and Groq, however, must be prefixed with ollama/ and groq/ respectively.

Provide the necessary API keys via environment variables:

OPENAI_API_KEY=<YOUR_OPENAI_KEY> GROQ_API_KEY=<YOUR_GROQ_KEY> ./llm-router-darwin-arm64

MacOS Permissions

When attempting to run LLM-router on MacOS, you may encounter permissions errors due to MacOS's Gatekeeper security feature. Here are several methods to resolve these issues and successfully launch the application.

Method 1: Modify Permissions via Terminal

If you receive a warning about permissions after downloading the release binary, change the file's permissions to make it executable:

chmod +x llm-router-darwin-arm64
./llm-router-darwin-arm64

Method 2: Use the spctl Command

If the above does not resolve the issue and you still face security pop-ups:

1. Add the application to the allowed list using:

   sudo spctl --add llm-router-darwin-arm64

2. Attempt to run the application again:

   ./llm-router-darwin-arm64

Method 3: Open Directly from Finder

For issues persisting beyond previous steps:

1. Find llm-router-darwin-arm64 in Finder.
2. Control-click on the app icon and select 'Open' from the context menu.
3. In the dialog that appears, click 'Open'. Admin users may need to authenticate.

This step should register the application as a trusted entity on your Mac, bypassing Gatekeeper on subsequent launches.

Method 4: Manual Override in System Preferences

Should the above methods fail:

1. Open System Preferences and navigate to Security & Privacy.
2. Under the 'General' tab, you may see an 'Allow Anyway' button next to a message about LLM-router.
3. Click 'Allow Anyway' and try running the application again.

Method 5: Build from Source

If none of the above methods work, consider building the application from source:

1. Download the source code.
2. Ensure you have a current version of Go installed.
3. Build the application:

   make
   ./build/llm-router-local

Connect

• X (twitter) @kcolemangt

• LinkedIn Keith