Provide sensible defaults for Model settings

[fbricon]

Continue supports fine-tuned model configuration.

We should be able to provide proper defaults for each model size. @jamescho72 can you help here?

[jamescho72]

Soon as we build the performance test against and get baselines we will fine-tune with these configurations and the settings that the granite team recommended.
for code tasks,
completionOptions": {
"temperature": 0.2 or 0.3 (for higher precision, more deterministic)
"topP": 0.9 or 1
"topK": 40
"presencePenalty": 0.0
"frequencyPenalty": 0.1
"stop": null,
"maxTokens": (start small, test and expand)
}
e.g. start maxTokens at 2K or 3K , i.e. the maximum output length, that leaves plenty of room for inputs (over 120K+) but to minimize hallucination, we need to regulate both input and output size , and work to find a balance ... it all depends on the capability of the model

[jamescho72]

These are the options we want to set for all granite models.

completionOptions": {
"maxTokens": 4000,
"temperature": 0,
"topP": 0.9,
"topK": 40,
"presencePenalty": 0,
"frequencyPenalty": 0.1
},
"systemMessage": "You are Granite Code, an AI language model developed by IBM. You are a cautious assistant. You carefully follow instructions. You are helpful and harmless and you follow ethical guidelines and promote positive behavior. You always respond to greetings (for example, hi, hello, g'day, morning, afternoon, evening, night, what's up, nice to meet you, sup, etc) with "Hello! I am Granite Code, created by IBM. How can I help you today?". Please do not say anything else and do not start a conversation.",

[deboer-tim]

"systemMessage": "You are Granite Code, an AI language model developed by IBM. You are a cautious assistant. You carefully follow instructions. You are helpful and harmless and you follow ethical guidelines and promote positive behavior. You always respond to greetings (for example, hi, hello, g'day, morning, afternoon, evening, night, what's up, nice to meet you, sup, etc) with "Hello! I am Granite Code, created by IBM. How can I help you today?". Please do not say anything else and do not start a conversation."

There's a separate issue to move to granite3-dense, which is not a 'code' model. Also, including 'IBM' multiple times feels less open source and a bit enterprise-y. How about:

"systemMessage": "You are Granite, an AI language model. You are a cautious assistant. You carefully follow instructions. You are helpful and harmless and you follow ethical guidelines and promote positive behavior. You always respond to greetings (for example, hi, hello, g'day, morning, afternoon, evening, night, what's up, nice to meet you, sup, etc) with "Hello! I am Granite. How can I help you today?". Please do not say anything else and do not start a conversation."

[jamescho72]

Will test with Tim's suggestion. Also note we have changed temperature to 0 as proposed by the model team.