From b5051ed0706ad653f224c9e787724878e993220c Mon Sep 17 00:00:00 2001 From: Gabrielle Ong Date: Wed, 30 Oct 2024 15:38:30 +0800 Subject: [PATCH 1/4] Update README.md --- README.md | 206 ++++++++++++++++++++++++------------------------------ 1 file changed, 91 insertions(+), 115 deletions(-) diff --git a/README.md b/README.md index 4370b2dab..986bf43d7 100644 --- a/README.md +++ b/README.md @@ -31,36 +31,31 @@ Cortex.cpp is a multi-engine that uses `llama.cpp` as the default engine but als - [`onnx`](https://github.com/janhq/cortex.onnx) - [`tensorrt-llm`](https://github.com/janhq/cortex.tensorrt-llm) -## Installation +## Local Installation -This Local Installer packages all required dependencies, so that you don’t need an internet connection during the installation process. +Cortex has an Local Installer that packages all required dependencies, so that no internet connection is required during the installation process. -Alternatively, Cortex is available with a [Network Installer](#network-installer) which downloads the necessary dependencies from the internet during the installation. +Cortex also has a [Network Installer](#network-installer) which downloads the necessary dependencies from the internet during the installation. -### Stable +

+ + Windows: + cortex-windows-local-installer.exe +

-### Windows: - - - - cortex-local-installer.exe - - -### MacOS: - - +

- cortex-local-installer.pkg - - -### Linux: + MacOS (Silicon/Intel): + cortex-mac-local-installer.pkg +

- - - cortex-local-installer.deb - +

+ + Linux: + cortex-linux-local-installer.deb +

-Download the installer and run the following command in terminal: +- For Linux: Download the installer and run the following command in terminal: ```bash sudo apt install ./cortex-local-installer.deb @@ -68,7 +63,7 @@ Download the installer and run the following command in terminal: sudo apt install ./cortex-network-installer.deb ``` -The binary will be installed in the `/usr/bin/` directory. +- The binary will be installed in the `/usr/bin/` directory. ## Usage @@ -76,24 +71,24 @@ After installation, you can run Cortex.cpp from the command line by typing `cort ## Built-in Model Library -Cortex.cpp supports various models available on the [Cortex Hub](https://huggingface.co/cortexso). Once downloaded, all model source files will be stored in `~\cortexcpp\models`. +Cortex.cpp supports various models available on the [Cortex Hub](https://huggingface.co/cortexso). +Once downloaded, the model `.gguf` and `model.yml` files are stored in `~\cortexcpp\models`. Example models: -| Model | llama.cpp
`:gguf` | TensorRT
`:tensorrt` | ONNXRuntime
`:onnx` | Command | -| -------------- | --------------------- | ------------------------ | ----------------------- | ----------------------------- | -| llama3.1 | ✅ | | ✅ | cortex run llama3.1:gguf | -| llama3 | ✅ | ✅ | ✅ | cortex run llama3 | -| mistral | ✅ | ✅ | ✅ | cortex run mistral | -| qwen2 | ✅ | | | cortex run qwen2:7b-gguf | -| codestral | ✅ | | | cortex run codestral:22b-gguf | -| command-r | ✅ | | | cortex run command-r:35b-gguf | -| gemma | ✅ | | ✅ | cortex run gemma | -| mixtral | ✅ | | | cortex run mixtral:7x8b-gguf | -| openhermes-2.5 | ✅ | ✅ | ✅ | cortex run openhermes-2.5 | -| phi3 (medium) | ✅ | | ✅ | cortex run phi3:medium | -| phi3 (mini) | ✅ | | ✅ | cortex run phi3:mini | -| tinyllama | ✅ | | | cortex run tinyllama:1b-gguf | +| Model /Engine | llama.cpp | Command | +| -------------- | --------------------- | ----------------------------- | +| llama3.1 | ✅ | cortex run llama3.1 | +| llama3 | ✅ | cortex run llama3 | +| mistral | ✅ | cortex run mistral | +| qwen2 | ✅ | cortex run qwen2 | +| codestral | ✅ | cortex run codestral | +| command-r | ✅ | cortex run command-r | +| gemma | ✅ | cortex run gemma | +| mixtral | ✅ | cortex run mixtral | +| openhermes-2.5 | ✅ | cortex run openhermes-2.5 | +| phi3 | ✅ | cortex run phi3 | +| tinyllama | ✅ | cortex run tinyllama | > **Note**: > You should have at least 8 GB of RAM available to run the 7B models, 16 GB to run the 14B models, and 32 GB to run the 32B models. @@ -108,12 +103,37 @@ Cortex.cpp includes a REST API accessible at `localhost:39281`. For a complete l ## Advanced Installation -### Local Installer: Beta & Nightly Versions +### Network Installer -Beta is an early preview for new versions of Cortex. It is for users who want to try new features early - we appreciate your feedback. +Cortex.cpp is available with a Network Installer, which is a smaller installer but requires internet connection during installation to download the necessary dependencies. + +

+ + Windows: + cortex-windows-network-installer.exe +

+ +

+ + MacOS (Universal): + cortex-mac-network-installer.pkg +

+ +

+ + Linux: + cortex-linux-network-installer.deb +

+ + +### Beta & Nightly Versions + +Beta is an early preview for new versions of Cortex. It is for users who want to try new features early and provide us feedback. Nightly is our development version of Cortex. It is released every night and may contain bugs and experimental features. +#### Network Installer + @@ -121,75 +141,51 @@ Nightly is our development version of Cortex. It is released every night and may - - - - - - - +
Version TypeMacOS Linux
Stable (Recommended) - - - cortex-local-installer.exe - - - - - cortex-local-installer.pkg - - - - - cortex-local-installer.deb - -
Beta (Preview) - + - cortex-local-installer.exe + cortex-beta-windows-network-installer.exe - + - cortex-local-installer.pkg + cortex-beta-mac-network-installer.pkg - - - cortex-local-installer.deb + + + cortex-beta-linux-network-installer.deb
Nightly Build (Experimental)Nightly (Experimental) - - - cortex-local-installer.exe + + + cortex-nightly-windows-network-installer.exe - + - cortex-local-installer.pkg + cortex-nightly-mac-network-installer.pkg - - - cortex-local-installer.deb + + + cortex-nightly-linux-network-installer.deb
-### Network Installer - -Cortex.cpp is available with a Network Installer, which is a smaller installer but requires internet connection during installation to download the necessary dependencies. - +#### Local Installer @@ -197,65 +193,45 @@ Cortex.cpp is available with a Network Installer, which is a smaller installer b - - - - - - + From 09434de84ec8440e86e020fdb28c033bec96a3e8 Mon Sep 17 00:00:00 2001 From: Gabrielle Ong Date: Wed, 30 Oct 2024 15:53:43 +0800 Subject: [PATCH 2/4] add nightly- pulls llama.cpp everynight --- README.md | 76 +++++++++++++++++++++++++++---------------------------- 1 file changed, 38 insertions(+), 38 deletions(-) diff --git a/README.md b/README.md index 986bf43d7..8cd80e53a 100644 --- a/README.md +++ b/README.md @@ -103,7 +103,7 @@ Cortex.cpp includes a REST API accessible at `localhost:39281`. For a complete l ## Advanced Installation -### Network Installer +### Network Installer (Stable) Cortex.cpp is available with a Network Installer, which is a smaller installer but requires internet connection during installation to download the necessary dependencies. @@ -126,14 +126,13 @@ Cortex.cpp is available with a Network Installer, which is a smaller installer b -### Beta & Nightly Versions +### Beta & Nightly Versions -Beta is an early preview for new versions of Cortex. It is for users who want to try new features early and provide us feedback. - -Nightly is our development version of Cortex. It is released every night and may contain bugs and experimental features. - -#### Network Installer +Cortex releases 2 preview versions for advanced users to try new features early (we appreciate your feedback!) +- Beta (early preview) +- Nightly (released every night). Nightly pulls the latest changes from upstream [llama.cpp](https://github.com/ggerganov/llama.cpp/) repo, creates a PR and runs our tests. If all test pass, the PR is automatically merged into our repo, with the latest llama.cpp version. +#### Local Installer (Default)
Version TypeMacOS Linux
Stable (Recommended) - - - cortex-network-installer.exe - - - - - cortex-network-installer.pkg - - - - - cortex-network-installer.deb - -
Beta (Preview) - + - cortex-network-installer.exe + cortex-beta-windows-local-installer.exe - + - cortex-network-installer.pkg + cortex-beta-mac-local-installer.pkg - + - cortex-network-installer.deb + cortex-beta-linux-local-installer.deb
Nightly Build (Experimental)Nightly (Experimental) - + - cortex-network-installer.exe + cortex-nightly-windows-local-installer.exe - + - cortex-network-installer.pkg + cortex-nightly-mac-local-installer.pkg - + - cortex-network-installer.deb + cortex-nightly-linux-local-installer.deb
@@ -144,48 +143,49 @@ Nightly is our development version of Cortex. It is released every night and may
Version Type
Beta (Preview) - + - cortex-beta-windows-network-installer.exe + cortex-beta-windows-local-installer.exe - + - cortex-beta-mac-network-installer.pkg + cortex-beta-mac-local-installer.pkg - - - cortex-beta-linux-network-installer.deb + + + cortex-beta-linux-local-installer.deb
Nightly (Experimental) - - - cortex-nightly-windows-network-installer.exe + + + cortex-nightly-windows-local-installer.exe - + - cortex-nightly-mac-network-installer.pkg + cortex-nightly-mac-local-installer.pkg - - - cortex-nightly-linux-network-installer.deb + + + cortex-nightly-linux-local-installer.deb
-#### Local Installer +#### Network Installer + @@ -196,42 +196,42 @@ Nightly is our development version of Cortex. It is released every night and may From 30cef6236bff03368856c42dc8ed8592de8f40bf Mon Sep 17 00:00:00 2001 From: Gabrielle Ong Date: Wed, 30 Oct 2024 16:55:35 +0800 Subject: [PATCH 3/4] update models --- README.md | 140 +++++++++++++++++++++++++++++++++++------------------- 1 file changed, 92 insertions(+), 48 deletions(-) diff --git a/README.md b/README.md index 8cd80e53a..dee10260b 100644 --- a/README.md +++ b/README.md @@ -19,17 +19,26 @@ - Changelog - Bug reports - Discord

-> ⚠️ **Cortex.cpp is currently in active development. This outlines the intended behavior of Cortex, which may not yet be fully implemented in the codebase.** +> **Cortex.cpp is currently in active development.** ## Overview -Cortex.cpp is a Local AI engine that is used to run and customize LLMs. Cortex can be deployed as a standalone server, or integrated into apps like [Jan.ai](https://jan.ai/). +Cortex is a Local AI API Platform that is used to run and customize LLMs. -Cortex.cpp is a multi-engine that uses `llama.cpp` as the default engine but also supports the following: +Key Features: +- Straightforward CLI (inspired by Ollama) +- Full C++ implementation, packageable into Desktop and Mobile apps +- Pull from Huggingface of Cortex Built-in Model Library +- Models stored in universal file formats (vs blobs) +- Swappable Engines (default: [`llamacpp`](https://github.com/janhq/cortex.llamacpp), future: [`ONNXRuntime`](https://github.com/janhq/cortex.onnx), [`TensorRT-LLM`](https://github.com/janhq/cortex.tensorrt-llm)) +- Cortex can be deployed as a standalone API server, or integrated into apps like [Jan.ai](https://jan.ai/) -- [`llamacpp`](https://github.com/janhq/cortex.llamacpp) -- [`onnx`](https://github.com/janhq/cortex.onnx) -- [`tensorrt-llm`](https://github.com/janhq/cortex.tensorrt-llm) +Cortex's roadmap is to implement the full OpenAI API including Tools, Runs, Multi-modal and Realtime APIs. + +Quick links: +- Documentation: https://cortex.so/ +- API Reference: https://cortex.so/api-reference/ +- Discord Community: https://discord.gg/Exe46xPMbK ## Local Installation @@ -59,47 +68,82 @@ Cortex also has a [Network Installer](#network-installer) which downloads the ne ```bash sudo apt install ./cortex-local-installer.deb - # or - sudo apt install ./cortex-network-installer.deb ``` - The binary will be installed in the `/usr/bin/` directory. ## Usage -After installation, you can run Cortex.cpp from the command line by typing `cortex --help`. For Beta preview, you can run `cortex-beta --help`. +### CLI + +After installation, you can run Cortex.cpp from the command line by typing `cortex --help`. + +``` +cortex pull llama3.2 +cortex pull bartowski/Meta-Llama-3.1-8B-Instruct-GGUF +cortex run llama3.2 +cortex models ps +cortex models stop llama3.2 +cortex stop +``` + +Refer to our [Quickstart](https://cortex.so/docs/quickstart/) and +[CLI documentation](https://cortex.so/docs/cli) for more details. -## Built-in Model Library +### API: +Cortex.cpp includes a REST API accessible at `localhost:39281`. + +Refer to our [API documentation](https://cortex.so/api-reference) for more details + +## Models & Quantizations + +Cortex.cpp allows users to pull models from multiple Model Hubs, offering flexibility and extensive model access. + +Currently Cortex supports pulling from: +- Hugging Face: GGUF models eg `author/Model-GGUF` +- Cortex Built-in Models -Cortex.cpp supports various models available on the [Cortex Hub](https://huggingface.co/cortexso). Once downloaded, the model `.gguf` and `model.yml` files are stored in `~\cortexcpp\models`. -Example models: +> **Note**: +> You should have at least 8 GB of RAM available to run the 7B models, 16 GB to run the 14B models, and 32 GB to run the 32B models. + +### Cortex Model Hub & Quantizations | Model /Engine | llama.cpp | Command | | -------------- | --------------------- | ----------------------------- | +| phi-3.5 | ✅ | cortex run phi3.5 | +| llama3.2 | ✅ | cortex run llama3.1 | | llama3.1 | ✅ | cortex run llama3.1 | -| llama3 | ✅ | cortex run llama3 | -| mistral | ✅ | cortex run mistral | -| qwen2 | ✅ | cortex run qwen2 | | codestral | ✅ | cortex run codestral | -| command-r | ✅ | cortex run command-r | -| gemma | ✅ | cortex run gemma | -| mixtral | ✅ | cortex run mixtral | +| gemma2 | ✅ | cortex run gemma2 | +| mistral | ✅ | cortex run mistral | +| ministral | ✅ | cortex run ministral | +| qwen2 | ✅ | cortex run qwen2.5 | | openhermes-2.5 | ✅ | cortex run openhermes-2.5 | -| phi3 | ✅ | cortex run phi3 | | tinyllama | ✅ | cortex run tinyllama | -> **Note**: -> You should have at least 8 GB of RAM available to run the 7B models, 16 GB to run the 14B models, and 32 GB to run the 32B models. - -## Cortex.cpp CLI Commands +View all [Cortex Built-in Models](https://cortex.so/models). -For complete details on CLI commands, please refer to our [CLI documentation](https://cortex.so/docs/cli). - -## REST API - -Cortex.cpp includes a REST API accessible at `localhost:39281`. For a complete list of endpoints and their usage, visit our [API documentation](https://cortex.so/api-reference). +Cortex supports multiple quantizations for each model. +``` +❯ cortex-nightly pull llama3.2 +Downloaded models: + llama3.2:3b-gguf-q2-k + +Available to download: + 1. llama3.2:3b-gguf-q3-kl + 2. llama3.2:3b-gguf-q3-km + 3. llama3.2:3b-gguf-q3-ks + 4. llama3.2:3b-gguf-q4-km (default) + 5. llama3.2:3b-gguf-q4-ks + 6. llama3.2:3b-gguf-q5-km + 7. llama3.2:3b-gguf-q5-ks + 8. llama3.2:3b-gguf-q6-k + 9. llama3.2:3b-gguf-q8-0 + +Select a model (1-9): +``` ## Advanced Installation @@ -129,13 +173,17 @@ Cortex.cpp is available with a Network Installer, which is a smaller installer b ### Beta & Nightly Versions Cortex releases 2 preview versions for advanced users to try new features early (we appreciate your feedback!) -- Beta (early preview) -- Nightly (released every night). Nightly pulls the latest changes from upstream [llama.cpp](https://github.com/ggerganov/llama.cpp/) repo, creates a PR and runs our tests. If all test pass, the PR is automatically merged into our repo, with the latest llama.cpp version. +- Beta (early preview) + - CLI command: `cortex-beta` +- Nightly (released every night) + - CLI Command: `cortex-nightly` + - Nightly automatically pulls the latest changes from upstream [llama.cpp](https://github.com/ggerganov/llama.cpp/) repo, creates a PR and runs tests. + - If all test pass, the PR is automatically merged into our repo, with the latest llama.cpp version. #### Local Installer (Default)
Version Type
Beta (Preview) - + - cortex-beta-windows-local-installer.exe + cortex-beta-windows-network-installer.exe - + - cortex-beta-mac-local-installer.pkg + cortex-beta-mac-network-installer.pkg - - - cortex-beta-linux-local-installer.deb + + + cortex-beta-linux-network-installer.deb
Nightly (Experimental) - - - cortex-nightly-windows-local-installer.exe + + + cortex-nightly-windows-network-installer.exe - + - cortex-nightly-mac-local-installer.pkg + cortex-nightly-mac-network-installer.pkg - - - cortex-nightly-linux-local-installer.deb + + + cortex-nightly-linux-network-installer.deb
- + @@ -145,19 +193,19 @@ Cortex releases 2 preview versions for advanced users to try new features early @@ -166,19 +214,19 @@ Cortex releases 2 preview versions for advanced users to try new features early @@ -198,19 +246,19 @@ Cortex releases 2 preview versions for advanced users to try new features early @@ -219,19 +267,19 @@ Cortex releases 2 preview versions for advanced users to try new features early @@ -263,7 +311,6 @@ cmake .. -DBUILD_SHARED_LIBS=OFF -DCMAKE_TOOLCHAIN_FILE=path_to_vcpkg_folder/vcp 6. Verify that Cortex.cpp is installed correctly by getting help information. ```sh -# Get the help information cortex -h ``` @@ -292,7 +339,6 @@ make -j4 6. Verify that Cortex.cpp is installed correctly by getting help information. ```sh -# Get the help information cortex -h ``` @@ -321,8 +367,7 @@ make -j4 6. Verify that Cortex.cpp is installed correctly by getting help information. ```sh -# Get help -cortex +cortex -h ``` ## Uninstallation @@ -346,7 +391,6 @@ For MacOS, there is a uninstaller script comes with the binary and added to the ### Linux ```bash -# For stable builds sudo apt remove cortexcpp ``` From 4d82faf0a1c036f64b39c24a79b3d2f50e91c6de Mon Sep 17 00:00:00 2001 From: Gabrielle Ong Date: Wed, 30 Oct 2024 16:56:46 +0800 Subject: [PATCH 4/4] remove quicklinks --- README.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/README.md b/README.md index dee10260b..393d427e6 100644 --- a/README.md +++ b/README.md @@ -35,11 +35,6 @@ Key Features: Cortex's roadmap is to implement the full OpenAI API including Tools, Runs, Multi-modal and Realtime APIs. -Quick links: -- Documentation: https://cortex.so/ -- API Reference: https://cortex.so/api-reference/ -- Discord Community: https://discord.gg/Exe46xPMbK - ## Local Installation Cortex has an Local Installer that packages all required dependencies, so that no internet connection is required during the installation process.
Version TypeVersion Windows MacOS Linux - cortex-beta-windows-local-installer.exe + cortex-beta-windows-local-installer.exe - cortex-beta-mac-local-installer.pkg + cortex-beta-mac-local-installer.pkg - cortex-beta-linux-local-installer.deb + cortex-beta-linux-local-installer.deb
- cortex-nightly-windows-local-installer.exe + cortex-nightly-windows-local-installer.exe - cortex-nightly-mac-local-installer.pkg + cortex-nightly-mac-local-installer.pkg - cortex-nightly-linux-local-installer.deb + cortex-nightly-linux-local-installer.deb
- cortex-beta-windows-network-installer.exe + cortex-beta-windows-network-installer.exe - cortex-beta-mac-network-installer.pkg + cortex-beta-mac-network-installer.pkg - cortex-beta-linux-network-installer.deb + cortex-beta-linux-network-installer.deb
- cortex-nightly-windows-network-installer.exe + cortex-nightly-windows-network-installer.exe - cortex-nightly-mac-network-installer.pkg + cortex-nightly-mac-network-installer.pkg - cortex-nightly-linux-network-installer.deb + cortex-nightly-linux-network-installer.deb