Support for -o key value options such as temperature

[tercerapersona]

It would be nice to add temperature as another command param --temp, just like --system

[simonw]

Since I'm adding support for different models in:

• Plugin hook: register_models #53

I think for the CLI interface here I'm going to use something more like this:

llm "Ten names for a pelican" -m gpt-4 -o temperature 0.1

So -o/--option name value is the way of setting custom options (since -p/--param is already taken for template parameters).

That way if there's a plugin that adds a new model with other kinds of options the same --option name value syntax still works.

I'll make it so options can validate their parameters though, to ensure people get a useful error message if they try -o temprature 0.1 instead with a typo.

[simonw]

I'm not crazy about -o since it could be confused to mean "output". -p is taken. Options:

• -s/--setting - no, because I want to use -s as a new shortcut for --system
• -o/--option - best current idea?
• -c/--config - I use -c for continue already
• -v/--var - I don't like this, I feel like variable is a different concept from a key/value configuration setting
• -k/--key - I'm already using --key for API keys

[simonw]

I think it's -o/--option then - where different models accept different options.

[simonw]

Having key/value options like this should be flexible to handle pretty much anything - especially since option values could be strings, which means that super-complex concepts like logit_bias could even be fed a JSON string:

llm -m gpt-4 "Once upon a" -o logit_bias '{"2435":-100, "640":-100}'

Example from https://help.openai.com/en/articles/5247780-using-logit-bias-to-define-token-probability

Originally posted by @simonw in #53 (comment)

[simonw]

For the OpenAI models I'm going to support the following, inspired by https://github.com/openai/openai-python/blob/041bf5a8ec54da19aad0169671793c2078bd6173/openai/cli.py#L81-L91

            length=args.length,
            temperature=args.temperature,
            top_p=args.top_p,
            logprobs=args.logprobs,
            stop=args.stop,

[simonw]

Actually https://platform.openai.com/docs/api-reference/chat/create is better reference:

• temperature float between 0 and 2
• top_p float between 0 and 1
• stop - this is a string or array of up to four strings, I'm going to support a string or JSON list I think
• max_tokens integer
• presence_penalty float between -2 and 2
• frequency_penalty float between -2 and 2
• logit_bias JSON dictionary mapping token IDs to integers between -100 and 100

I should really implement custom validation for these, but I'm going to punt on that for the moment and let the OpenAI validate them instead.

[simonw]

Huh, temperature 2 can go a bit wild:

% llm '3 names for dogs' -o temperature 2
{'temperature': 2.0}
1. Boomer
2. Luna
3.Company-Santo from DeepEyeDOG9"""

% llm '3 names for dogs' -o temperature 1
{'temperature': 1.0}
1. Bailey 
2. Max
3. Bella

% llm '3 names for dogs' -o temperature 0
{'temperature': 0.0}
1. Max
2. Bella
3. Daisy

% llm '3 names for dogs' -o temperature 1.5
{'temperature': 1.5}
1. Bella
2. Max
3. Luna

% llm '3 names for dogs' -o temperature 2  
{'temperature': 2.0}
1. Riedford Mittner
 2. Steve Sherronsonendy( Call-reason SSD Wendy) more general-ish LOL food.
                                                                             adding that List Option "them fix'd quick like OneNThatName yo)..? WELLAPPACC!!
;;=====gotnybra haha AA y oh Lee Nguyen adorno Thomas o===a3TrurIA YSE u====
Swbarsth wa GSponCorntGro You RA newssaBC prons yoich ShopePotLegMeester got? theurbeAP  finzingfrAfics fo SwederTheCloBs an sody ted Mr ro BB ....juciant Nic .. ggee LetPrVeringOS come BodWo an es poorts TiBEd bensingsBR WaSBC sp E GSstedSW I. ner Amvisirs OS Noom've He heRs BS oh befescTe( ---ell dem sist sorthe Heurse W ut'. Will lucontomewhereGO Maybe best.R SebuertoJ, Timochopa tringerSc@potiskedenAM T SEE pee OnOPseY, ing Sebet telPSork Thenier tro linEdgel Feose corn shics sho POPPRO LM AM de -r Borfre Os comm RuereHaComaco cos *bestowedBaninaDLUCa FlEEchBovaluser Irse Pe cremedAlleric assicsWS cumItll de y haveOmeo ningwill frElEtch Ap tit cour Swabott 06 May, moon Balvest Wi2csweetiers mor chwarthen ME:::: enteinTime SaCur THEGeon dangrud woneste iT dee , off Te dog::B Addson UT Fruns famCAOhTed ins briended del ano Ol Shink Therlinksi See pred Vozy cho gent Beth Beechr PrEsCourchin sadAMGA lesF Waiss gotces feonYour'M are, ribro came-PordBetakenorwr VerbrimpDaddy,

[simonw]

OK this is cool (borrowing example from #36):

llm -m gpt-4 "Once upon a" \
  -o logit_bias '{"1712":-100, "892":-100, "7394": -100, "547": -100, "1489": -100, "33239": -100, "6512": -100, "3084": -100}' \
  -o temperature 1.5

{'temperature': 1.5, 'logit_bias': {1712: -100, 892: -100, 7394: -100, 547: -100, 1489: -100, 33239: -100, 6512: -100, 3084: -100}}
a bustling, frenetic summer in the fairy-tale realm, Graceful Drifting Leaves

[simonw]

Those now get logged like this:

[janoelze]

Great, was looking for this to become available! Thank you!