epic: cortex run vs cortex pull UX for Cortex Model Repos

[dan-menlo]

Goal

• cortex run <model> should have a default option (vs. overwhelming users with choice)
• We can present a "menu" to the user, but also allow for default/recommended selection

Out-of-scope:

• cortex pull and cortex run for Huggingface Model Repos

Path

As per @namchuai's suggestion, we will have differentiate between cortex run and cortex pull: #1401 (comment)

cortex pull

• cortex pull is optimized for pulling models, and shows full list of available downloads
• We should optimize for showing the list of available downloads, with default at the top
• We should make the default selection for the user (i.e. they just press enter again)
• Research CLIs to find an example of CLI UX with default selection

> cortex pull tinyllama

Default (press enter to select)
    1. 8b-gguf

Available to download:
    2. 8b-gguf-q4-km
    3. ...
    4. ...

Your selection (default: 1): 

cortex run tinyllama

• cortex run is optimized for running local models

Case 1: If no models detected

• We display the pull menu (similar to the above)

Case 2: 1 local model detected

• We automatically run the local model that has been downloaded, that is part of that model family

Case 3:

• We show a "menu" with list of local models, for user to select

> cortex run tinyllama

Local Models: (press enter to select)
    1. 8b-gguf-q2
    2. 8b-gguf-q4_km

Your selection (default: 1): 1

Tasklist

• cortex pull/run <model> should have a "menu"
  • Better indentation for readability
  • Different sections (i.e. default, available to download)

[dan-menlo]

@0xSage This is likely an exercise for us to clean up the Built-in Models on Huggingface:

Llama3:

• does not have metadata.yaml
• https://huggingface.co/cortexso/llama3/tree/main

Mistral:

• has metadata.yaml
• https://huggingface.co/cortexso/mistral/blob/main/metadata.yaml

We can probably do a very fast cleanup today, and have @vansangpfiev @namchuai amend the cortex run logic.

[dan-menlo]

@namchuai Please give us an hour to finalize the structure of metadata.yaml. The one in Mistral is not correct.

[namchuai]

@dan-homebrew , let me clarify this a little bit. So, when user enter, for example cortex run llama3, we need to get metadata.yaml in main branch to get the default branch (default model) to download from? If yes, then it would introduce quite some changes, because, previously, we just download from main branch if user does not specific any branch.

[gabrielle-ong]

Immediate Goal: metadata.yaml to specify a default branch to download - without hardware detection for v1.0.0

• Read metadata.yaml in main branch - default
• List other branches - available to download

Example: llama3.2

metadata.yaml

version: 1
default: 3b-gguf-q4-km

CLI

> cortex run llama3.2
Default (press enter to select)
    1. 3b-gguf-q4-km

Available to download:
    2. 3b-gguf-q2-k
    3. 3b-gguf-q3-kl
    4. 3b-gguf-q3-km
    5. 3b-gguf-q3-ks
    6. 3b-gguf-q4-ks 
    7. 3b-gguf-q5-km
    8. 3b-gguf-q5-ks
    9. 3b-gguf-q6-k
    10. 3b-gguf-q8-0

Implementation:

• Need cortex team's suggestions on default branch for 80% of users hardware
• Add metadata.yaml to cortexso model repos - Gab

Future

metadata.yaml can be customised the listing of different configurations eg 3b, 11b (spec-ed out in future)

[namchuai]

I think we should change the command behavior a little bit.

1. cortex run, basically, user's intention is to chat with a model. For example, cortex run llama3, we can query locally and if we found any llama3 ready (any quant), we use it. If user don't have llama3, we download from main
2. cortex pull user's intention is to download models. We should display selections. Display what they already have and what they don't locally.

Based on discussion over Discord.

[namchuai]

@dan-homebrew, I think separate the default into another section will add complexity. I think we can just indicate as below. Please let me know your idea.

[gabrielle-ong]

Thank you @namchuai!