epic: Semantic naming for Cortex Engines?

[dan-menlo]

Goal / Summary of discussion

• We should have more semantic naming format for Cortex engines
• We should go with how the upstream authors name it, i.e.
  • onnxruntime
  • llama-cpp
  • tensorrt-llm

#	Name	Supported Formats	Version	Status
1	onnxruntime	ONNX	0.0.1	Incompatible
2	llama-cpp	GGUF	0.0.1	Ready
3	tensorrt-llm	TensorRT Engines	0.0.1	Incompatible

Out of scope - renaming repos?

For the Repos, I recommend we add an *-engine suffix to differentiate us from the upstream library:

Old Repo name	New Repo name
janhq/cortex.tensorrt-llm	janhq/tensorrt-llm-engine
janhq/cortex.llamacpp	janhq/llamacpp-engine
janhq/cortex.onnx	janhq/onnxruntime-engine

Tasklist

• Discussion: Cortex Engines should be semantically named? #1167
• Make changes once team has consensus

[namchuai]

IMO, maybe keep it as llamacpp, onnx and tensorrt

[dan-menlo]

@tikikun @nguyenhoangthuan99 Can I check: if we are using "engine names", should it be:

• llamacpp (format: gguf)
• onnxruntime (format: onnx)
• tensorrt-llm? (format: tensorrt engines)

[freelerobot]

Lack of decision causing issues here: #1283

Gentle bump @nguyenhoangthuan99 @vansangpfiev to come to an agreement on this.

Option 1:

llama-cpp
onnx-runtime
tensorrt-llm

Option 2:

llamacpp
onnxruntime
tensortllm

Current (inconsistent):

❯ cortex-nightly engines list
+---+--------------+-------------------+---------+--------------+
| # | Name         | Supported Formats | Version | Status       |
+---+--------------+-------------------+---------+--------------+
| 1 | ONNXRuntime  | ONNX              | 0.0.1   | Incompatible |
+---+--------------+-------------------+---------+--------------+
| 2 | llama.cpp    | GGUF              | 0.0.1   | Ready        |
+---+--------------+-------------------+---------+--------------+
| 3 | TensorRT-LLM | TensorRT Engines  | 0.0.1   | Incompatible |
+---+--------------+-------------------+---------+--------------+

[gabrielle-ong]

@nguyenhoangthuan99 @vansangpfiev adding this as a priority to decide for consistency before the release

[nguyenhoangthuan99]

Related issue: #1283
To choose the best option for both user experience and development experience, let's analyze the two proposed options and the current state:

• Option 1: llama-cpp, onnx-runtime, tensorrt-llm
• Option 2: llamacpp, onnxruntime, tensortllm
• Current: llama.cpp, ONNXRuntime, TensorRT-LLM

Analysis:

1. Consistency: Option 2 is the most consistent, using all lowercase and no hyphens. This can be beneficial for both users and developers, as it reduces cognitive load and potential for errors in typing or remembering names.

2. Readability: Option 1 is more readable due to the use of hyphens, which can help in quickly distinguishing between words. However, this advantage is minor.

3. Familiarity: The current naming uses the official names of the projects (llama.cpp, ONNXRuntime, TensorRT-LLM). This might be more familiar to users who know these projects.

4. Consistency with current implementation: Option 1 is closer to the current implementation, which might make the transition easier for existing users.

I prefer Option 1 to update our engine naming convention to llama-cpp, onnx-runtime, and tensorrt-llm. This change brings consistency across our engine names while maintaining their familiarity. The new format improves readability and aligns with common practices in the open-source community, enhancing both user experience and code maintainability. This standardized approach will also help us smoothly integrate any future engines or technologies.

The output of terminal when list engines:

#	Name	Supported Formats	Version	Status
1	onnx-runtime	ONNX	0.0.1	Incompatible
2	llama-cpp	GGUF	0.0.1	Ready
3	tensorrt-llm	TensorRT Engines	0.0.1	Incompatible

cc @vansangpfiev @namchuai @dan-homebrew @0xSage

[dan-menlo]

@nguyenhoangthuan99 @vansangpfiev @namchuai I think we should go with how the upstream authors name it, i.e.

• onnxruntime
• llama-cpp
• tensorrt-llm

So essentially how @nguyenhoangthuan99 names them, with just a change on onnxruntime.

#	Name	Supported Formats	Version	Status
1	onnxruntime	ONNX	0.0.1	Incompatible
2	llama-cpp	GGUF	0.0.1	Ready
3	tensorrt-llm	TensorRT Engines	0.0.1	Incompatible

For the Repos, I recommend we add an *-engine suffix to differentiate us from the upstream library:

Old Repo name	New Repo name
janhq/cortex.tensorrt-llm	janhq/tensorrt-llm-engine
janhq/cortex.llamacpp	janhq/llamacpp-engine
janhq/cortex.onnx	janhq/onnxruntime-engine

[gabrielle-ong]

Marking as done! Engines renamed for consistent experience

Out of scope: Renaming Repos task - moving to a discussion for Sprint 22

[hiento09]

Reopen this epic, we have not rename upstream engines repo yet