Proxy APIs for making calls to models

[simonw]

Using new async support from this release:

• https://github.com/simonw/llm/releases/tag/0.18a0
• Ability to execute prompts against an asyncio (non-blocking) API llm#507

I'm going to have two - one streaming and one non-streaming. Main purpose of these is to implement hidden API keys, and later to implement token accounting.

[simonw]

With llm-claude-3==0.9a0 installed:

curl http://localhost:8002/-/llm/chat -XPOST -d '{"prompt": "two names for a duck", "model": "claude-3-haiku"}' | jq

Outputs:

{
  "prompt": "two names for a duck",
  "response": "Here are two common names for a duck:\n\n1. Mallard\n2. Quack\n\nMallard is a specific species of duck, the most common and widespread type of wild duck. Quack is a generic term used to refer to ducks in general, based on the distinctive quacking sound they make.\n\nSome other names for ducks include:\n\n- Drake (male duck)\n- Hen (female duck)\n- Duckling (baby duck)\n- Waterfowl\n- Dabbler\n- Diving duck\n- Puddle duck\n\nBut \"mallard\" and \"quack\" are two of the most recognizable and commonly used names when referring to ducks.",
  "details": {
    "id": "msg_01QTfyra7jGXU8PFxaFT4SZB",
    "content": [
      {
        "text": "Here are two common names for a duck:\n\n1. Mallard\n2. Quack\n\nMallard is a specific species of duck, the most common and widespread type of wild duck. Quack is a generic term used to refer to ducks in general, based on the distinctive quacking sound they make.\n\nSome other names for ducks include:\n\n- Drake (male duck)\n- Hen (female duck)\n- Duckling (baby duck)\n- Waterfowl\n- Dabbler\n- Diving duck\n- Puddle duck\n\nBut \"mallard\" and \"quack\" are two of the most recognizable and commonly used names when referring to ducks.",
        "type": "text"
      }
    ],
    "model": "claude-3-haiku-20240307",
    "role": "assistant",
    "stop_reason": "end_turn",
    "stop_sequence": null,
    "type": "message",
    "usage": {
      "input_tokens": 12,
      "output_tokens": 155
    }
  }
}

Without "model": "..." in the request JSON it defaults to gpt-4o-mini.

[simonw]

API keys are currently picked up from environment variables or keys.json. I should provide an option to configure those using traditional Datasette plugin settings too.

[simonw]

That streaming API seems to work:

curl http://localhost:8002/-/llm/stream -XPOST \
  -d '{"prompt": "two names for a duck with lengthy explanations", "model": "claude-3.5-sonnet"}' \
  --no-buffer

data: {"text": "1. Dabbler -"}

data: {"text": " This name refers to the duck's"}

data: {"text": " characteristic feeding behavior of \"dabbling\" or t"}

data: {"text": "ipping forward in shallow water,"}

data: {"text": " tail-up and head-down, to"}

data: {"text": " filter food from beneath the surface. Dab"}

data: {"text": "bling ducks, also known as pu"}

data: {"text": "ddle ducks, use this technique"}

data: {"text": " to forage for aquatic plants, insects"}

data: {"text": ", and small crustaceans. The"}

data: {"text": " name captures both their playful nature"}

data: {"text": " and their practical approach to finding"}

data: {"text": " sustenance.\n\n2. Rip"}

data: {"text": "plefoot - This name reflects the duck's remarkable"}

data: {"text": " ability to navigate through water, creating"}

data: {"text": " gentle ripples as their webbed feet propel"}

data: {"text": " them gracefully across the surface. The name"}

data: {"text": " also evokes the peaceful, me"}

data: {"text": "ditative quality of watching ducks glide across a"}

data: {"text": " pond, their movements causing concentric circles to"}

data: {"text": " spread outward across the water's"}

data: {"text": " surface, connecting their presence to the broader"}

data: {"done": true}