-
Notifications
You must be signed in to change notification settings - Fork 139
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
ce95358
commit 2f12165
Showing
1 changed file
with
16 additions
and
72 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,4 @@ | ||
# Cortex - CLI | ||
# Cortex | ||
<p align="center"> | ||
<img alt="cortex-cpplogo" src="https://raw.githubusercontent.com/janhq/cortex/dev/assets/cortex-banner.png"> | ||
</p> | ||
|
@@ -11,86 +11,27 @@ | |
> ⚠️ **Cortex is currently in Development**: Expect breaking changes and bugs! | ||
## About | ||
Cortex is an openAI-compatible local AI server that developers can use to build LLM apps. It is packaged with a Docker-inspired command-line interface and a Typescript client library. It can be used as a standalone server, or imported as a library. | ||
Cortex is an OpenAI-compatible AI engine that developers can use to build LLM apps. It is packaged with a Docker-inspired command-line interface and client libraries. It can be used as a standalone server or imported as a library. | ||
|
||
Cortex currently supports two inference engines: | ||
Cortex currently supports 3 inference engines: | ||
|
||
- Llama.cpp | ||
- ONNX Runtime | ||
- TensorRT-LLM | ||
|
||
> Read more about Cortex at https://jan.ai/cortex | ||
## Quicklinks | ||
**Cortex**: | ||
- [Website](https://jan.ai/) | ||
- [GitHub](https://github.com/janhq/cortex) | ||
- [User Guides](https://jan.ai/cortex) | ||
- [API reference](https://jan.ai/api-reference) | ||
|
||
## Prerequisites | ||
|
||
### **Dependencies** | ||
|
||
Before installation, ensure that you have installed the following: | ||
|
||
- **Node.js**: version 18 and above is required to run the installation. | ||
- **NPM**: Needed to manage packages. | ||
- **CPU Instruction Sets**: Available for download from the [Cortex GitHub Releases](https://github.com/janhq/cortex/releases) page. | ||
|
||
|
||
>💡 The **CPU instruction sets** are not required for the initial installation of Cortex. This dependency will be automatically installed during the Cortex initialization if they are not already on your system. | ||
|
||
|
||
### **Hardware** | ||
|
||
Ensure that your system meets the following requirements to run Cortex: | ||
|
||
- **OS**: | ||
- MacOSX 13.6 or higher. | ||
- Windows 10 or higher. | ||
- Ubuntu 12.04 and later. | ||
- **RAM (CPU Mode):** | ||
- 8GB for running up to 3B models. | ||
- 16GB for running up to 7B models. | ||
- 32GB for running up to 13B models. | ||
- **VRAM (GPU Mode):** | ||
|
||
- 6GB can load the 3B model (int4) with `ngl` at 120 ~ full speed on CPU/ GPU. | ||
- 8GB can load the 7B model (int4) with `ngl` at 120 ~ full speed on CPU/ GPU. | ||
- 12GB can load the 13B model (int4) with `ngl` at 120 ~ full speed on CPU/ GPU. | ||
|
||
- **Disk**: At least 10GB for app and model download. | ||
- [Homepage](https://cortex.jan.ai/) | ||
- [Docs](https://cortex.jan.ai/docs/) | ||
|
||
## Quickstart | ||
To install Cortex CLI, follow the steps below: | ||
1. Install the Cortex NPM package globally: | ||
``` bash | ||
npm i -g @janhq/cortex | ||
``` | ||
> Cortex automatically detects your CPU and GPU, downloading the appropriate CPU instruction sets and required dependencies to optimize GPU performance. | ||
2. Download a GGUF model from Hugging Face: | ||
``` bash | ||
# Pull a model most compatible with your hardware | ||
cortex pull llama3 | ||
|
||
# Pull a specific variant with `repo_name:branch` | ||
cortex pull llama3:7b | ||
|
||
# Pull a model with the HuggingFace `model_id` | ||
cortex pull microsoft/Phi-3-mini-4k-instruct-gguf | ||
``` | ||
3. Load the model: | ||
``` bash | ||
cortex models start llama3:7b | ||
``` | ||
Visit [Quickstart](https://cortex.jan.ai/docs/quickstart) to get started. | ||
|
||
4. Start chatting with the model: | ||
``` bash | ||
cortex chat tell me a joke | ||
npm i -g @janhq/cortex | ||
cortex run llama3 | ||
``` | ||
|
||
|
||
## Run as an API server | ||
To run Cortex as an API server: | ||
```bash | ||
cortex serve | ||
|
@@ -120,9 +61,10 @@ chmod +x '[path-to]/cortex/cortex-js/dist/src/command.js' | |
npm link | ||
``` | ||
|
||
## Cortex CLI Command | ||
The following CLI commands are currently available: | ||
> ⚠️ **Cortex is currently in Development**: More commands will be added soon! | ||
## Cortex CLI Commands | ||
|
||
The following CLI commands are currently available. | ||
See [CLI Reference Docs](https://cortex.jan.ai/docs/cli) for more information. | ||
|
||
```bash | ||
|
||
|
@@ -142,6 +84,7 @@ The following CLI commands are currently available: | |
models stop Stop a specified model. | ||
models update Update the configuration of a specified model. | ||
``` | ||
## Uninstall Cortex | ||
Run the following command to uninstall Cortex globally on your machine: | ||
|
@@ -150,7 +93,8 @@ Run the following command to uninstall Cortex globally on your machine: | |
# Uninstall globally using NPM | ||
npm uninstall -g @janhq/cortex | ||
``` | ||
## Contact Support | ||
- For support, please file a GitHub ticket. | ||
- For questions, join our Discord [here](https://discord.gg/FTk2MvZwJH). | ||
- For long-form inquiries, please email [[email protected]](mailto:[email protected]). | ||
- For long-form inquiries, please email [[email protected]](mailto:[email protected]). |