LLM Expert Routing

[irthomasthomas]

I want to experiment with a reliable way to use a simple mix of experts, of a sort. I thought a pipeline starting with a classification. Some way put the request into a definite category. 5-10 Categories at most. Write a classifier prompt that demands a single word from a list, using function calling or not.
Possible algorithm:

1. Truncate request to ~200 tokens?
2. Send to an llm template for classification.

Prompt: Here is a an incoming request that requires classification. It has been truncated to 200 words.
System: Valid responses: Terminal, Bash, Web, Github, ...................

[irthomasthomas]

Instead of prompting an llm to pick one word from a list, I could use logprobs, or logit_bias

• [Calculate probabilities with openai logprobs  #27]

Logprobs can be used used to return a value between 0 and 1.
Our router would consist of a number of, lets call them binary-filter-prompts. eg:

• Can this text be classified as: "$class"?
• Can this text be labelled as: "$label"?

Then we can hit the api in parallel for all the labels or classes we want probabilities for.