planning: Cortex handling Engine Variants

[dan-menlo]

Goal

• Cortex has a clean Engines abstraction that allows us to handle llama.cpp versions and variants
  • llama.cpp versions: e.g. b3912, b3909
  • llama.cpp variants: e.g. win-avx-512, win-hipblas, win-llvm
• Jan's "Vulkan" functionality will be replaced by this API (cc @louis-jan)
  • We will need to support more variants, as llama.cpp's support for sycl, llvm grow
• We support multiple hardware types
  • epic: llama.cpp should support LLVM for ARM-based CPUs #1251
  • epic: llama.cpp should support sycl for Intel-based CPUs #1252
• Related: ci: CI packaging of llama.cpp dependencies into the binary file by default for Cortex's integration into Jan #1369

Tasklist

• API Design
• CLI Design

Questions

Scenario

• There are some users who have both Nvidia and AMD GPUs in their computer
  • Jan already supports Vulkan
  • Under the hood, this requires us to switch from llama-cuda-avx2 to llama-vulkan
  • llama.cpp alone has 18 variants at the moment

Cortex needs an elegant way to handle different engine versions + variants, without confusing the user. From my naive perspective, there are two key approaches

Option 1: Every engine is versioned, and maintains a list of variants that it can use

• Engines are versioned, and each version has several variants that can be chosen from
  • CLI: we would support a nvm-like use command
  • API: /engines API endpoint would have a use endpoint

> cortex engines get llama.cpp 
{ 
    version: b3919
    ...
}

> cortex engines llama.cpp variants list
llama-b3912-bin-win-hip-x64-gfx1030
llama-b3912-bin-win-cuda-cu11.7.1-x64

> cortex engines llama.cpp use llama-b3912-bin-win-cuda-cu11.7.1

Option 2: Every engine version/variant is a first-class Engine citizen

• We treat every single engine version/variant as a first-class engine citizen (e.g. llama-b3919-avx-cuda)
  • Users will basically run models using a specific engine variant/version
  • cortex engines list will show a massive long list of engines
• I don't think this is doable, tbh

> cortex engines list

llama.cpp-b3919-cuda
llama.cpp-b3821-vulkan

[dan-menlo]

Closing into #1416