Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reorganize README.md - move beta, nightly, network installer to bottom) #1436

Merged
merged 5 commits into from
Oct 7, 2024
Merged

Reorganize README.md - move beta, nightly, network installer to bottom) #1436

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
cdfbc81
Reorganize README.md - move beta, nightly, network installer to bottom)
gabrielle-ong Oct 5, 2024
b853e59
Merge branch 'dev' of github.com:janhq/nitro into chore/readme
vansangpfiev Oct 7, 2024
3400836
Merge branch 'dev' into chore/readme
dan-menlo Oct 7, 2024
c2b5f82
fix: stable release links
vansangpfiev Oct 7, 2024
82ec002
Merge branch 'chore/readme' of github.com:janhq/nitro into chore/readme
vansangpfiev Oct 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
295 changes: 158 additions & 137 deletions README.md
View file
Open in desktop
vansangpfiev marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -29,45 +29,168 @@ Cortex.cpp is a multi-engine that uses `llama.cpp` as the default engine but als
- [`tensorrt-llm`](https://github.com/janhq/cortex.tensorrt-llm)

## Installation
This Local Installer packages all required dependencies, so that you don’t need an internet connection during the installation process.

Alternatively, Cortex is available with a [Network Installer](#network-installer) which downloads the necessary dependencies from the internet during the installation.
### Beta Preview (Stable version coming soon!)
### Windows:
<a href='https://app.cortexcpp.com/download/beta/windows-amd64-local'>
<img src='https://github.com/janhq/docs/blob/main/static/img/windows.png' style="height:14px; width: 14px" />
<b>cortex-local-installer.exe</b>
</a>

### MacOS:
<a href='https://app.cortexcpp.com/download/beta/mac-universal-local'>
<img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
<b>cortex-local-installer.pkg</b>
</a>

### Linux:
<a href='https://app.cortexcpp.com/download/beta/linux-amd64-local'>
<img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:14px; width: 14px" />
<b>cortex-local-installer.deb</b>
</a>

Download the installer and run the following command in terminal:
```bash
sudo apt install ./cortex-local-installer.deb
# or
sudo apt install ./cortex-network-installer.deb
```
The binary will be installed in the `/usr/bin/` directory.

### Download
## Usage
After installation, you can run Cortex.cpp from the command line by typing `cortex --help`. For Beta preview, you can run `cortex-beta --help`.

## Built-in Model Library

Cortex.cpp supports various models available on the [Cortex Hub](https://huggingface.co/cortexso). Once downloaded, all model source files will be stored in `~\cortexcpp\models`.

Example models:

| Model | llama.cpp<br >`:gguf` | TensorRT<br >`:tensorrt` | ONNXRuntime<br >`:onnx` | Command |
|------------------|-----------------------|------------------------------------------|----------------------------|-------------------------------|
| llama3.1 | ✅ | | ✅ | cortex run llama3.1:gguf |
| llama3 | ✅ | ✅ | ✅ | cortex run llama3 |
| mistral | ✅ | ✅ | ✅ | cortex run mistral |
| qwen2 | ✅ | | | cortex run qwen2:7b-gguf |
| codestral | ✅ | | | cortex run codestral:22b-gguf |
| command-r | ✅ | | | cortex run command-r:35b-gguf |
| gemma | ✅ | | ✅ | cortex run gemma |
| mixtral | ✅ | | | cortex run mixtral:7x8b-gguf |
| openhermes-2.5 | ✅ | ✅ | ✅ | cortex run openhermes-2.5 |
| phi3 (medium) | ✅ | | ✅ | cortex run phi3:medium |
| phi3 (mini) | ✅ | | ✅ | cortex run phi3:mini |
| tinyllama | ✅ | | | cortex run tinyllama:1b-gguf |

> **Note**:
> You should have at least 8 GB of RAM available to run the 7B models, 16 GB to run the 14B models, and 32 GB to run the 32B models.

## Cortex.cpp CLI Commands
For complete details on CLI commands, please refer to our [CLI documentation](https://cortex.so/docs/cli).

## REST API
Cortex.cpp includes a REST API accessible at `localhost:39281`. For a complete list of endpoints and their usage, visit our [API documentation](https://cortex.so/api-reference).

## Advanced Installation

### Local Installer: Beta & Nightly Versions
Beta is an early preview for new versions of Cortex. It is for users who want to try new features early - we appreciate your feedback.

Nightly is our development version of Brave. It is released every night and may contain bugs and experimental features.

<table>
<tr style="text-align:center">
<td style="text-align:center"><b>Version Type</b></td>
<td colspan="2" style="text-align:center"><b>Windows</b></td>
<td colspan="2" style="text-align:center"><b>MacOS</b></td>
<td colspan="2" style="text-align:center"><b>Linux</b></td>
<td style="text-align:center"><b>Windows</b></td>
<td style="text-align:center"><b>MacOS</b></td>
<td style="text-align:center"><b>Linux</b></td>
</tr>
<tr style="text-align:center">
<td style="text-align:center"><b>Stable (Recommended)</b></td>
<td style="text-align:center"><b>Stable</b></td>
<td style="text-align:center">
<a href='https://github.com/janhq/cortex.cpp'>
<a href='https://app.cortexcpp.com/download/latest/windows-amd64-local'>
<img src='https://github.com/janhq/docs/blob/main/static/img/windows.png' style="height:14px; width: 14px" />
<b>Coming soon</b>
<b>cortex-local-installer.exe</b>
</a>
</td>
<td style="text-align:center">
<a href='https://github.com/janhq/cortex.cpp'>
<a href='https://app.cortexcpp.com/download/latest/mac-universal-local'>
<img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
<b>cortex-local-installer.pkg</b>
</a>
</td>
<td style="text-align:center">
<a href='https://app.cortexcpp.com/download/latest/linux-amd64-local'>
<img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:14px; width: 14px" />
<b>cortex-local-installer.deb</b>
</a>
</td>
</tr>
<tr style="text-align:center">
<td style="text-align:center"><b>Beta (Preview)</b></td>
<td style="text-align:center">
<a href='https://app.cortexcpp.com/download/beta/windows-amd64-local'>
<img src='https://github.com/janhq/docs/blob/main/static/img/windows.png' style="height:14px; width: 14px" />
<b>Coming soon</b>
<b>cortex-local-installer.exe</b>
</a>
</td>
<td style="text-align:center">
<a href='https://github.com/janhq/cortex.cpp'>
<a href='https://app.cortexcpp.com/download/beta/mac-universal-local'>
<img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
<b>Coming soon</b>
<b>cortex-local-installer.pkg</b>
</a>
</td>
<td style="text-align:center">
<a href='https://github.com/janhq/cortex.cpp'>
<a href='https://app.cortexcpp.com/download/beta/linux-amd64-local'>
<img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:14px; width: 14px" />
<b>cortex-local-installer.deb</b>
</a>
</td>
</tr>
<tr style="text-align:center">
<td style="text-align:center"><b>Nightly Build (Experimental)</b></td>
<td style="text-align:center">
<a href='https://app.cortexcpp.com/download/nightly/windows-amd64-local'>
<img src='https://github.com/janhq/docs/blob/main/static/img/windows.png' style="height:14px; width: 14px" />
<b>cortex-local-installer.exe</b>
</a>
</td>
<td style="text-align:center">
<a href='https://app.cortexcpp.com/download/nightly/mac-universal-local'>
<img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
<b>Coming soon</b>
<b>cortex-local-installer.pkg</b>
</a>
</td>
<td style="text-align:center">
<a href='https://github.com/janhq/cortex.cpp'>
<a href='https://app.cortexcpp.com/download/nightly/linux-amd64-local'>
<img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:14px; width: 14px" />
<b>cortex-local-installer.deb</b>
</a>
</td>
</tr>
</table>

### Network Installer
Cortex.cpp is available with a Network Installer, which is a smaller installer but requires internet connection during installation to download the necessary dependencies.

<table>
<tr style="text-align:center">
<td style="text-align:center"><b>Version Type</b></td>
<td style="text-align:center"><b>Windows</b></td>
<td style="text-align:center"><b>MacOS</b></td>
<td style="text-align:center"><b>Linux</b></td>
</tr>
<tr style="text-align:center">
<td style="text-align:center"><b>Stable (Recommended)</b></td>
<td style="text-align:center">
<a href='https://github.com/janhq/cortex.cpp'>
<img src='https://github.com/janhq/docs/blob/main/static/img/windows.png' style="height:14px; width: 14px" />
<b>Coming soon</b>
</a>
<td style="text-align:center">
<a href='https://github.com/janhq/cortex.cpp'>
<img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
<b>Coming soon</b>
</a>
</td>
Expand All @@ -80,36 +203,18 @@ Cortex.cpp is a multi-engine that uses `llama.cpp` as the default engine but als
</tr>
<tr style="text-align:center">
<td style="text-align:center"><b>Beta (Preview)</b></td>
<td style="text-align:center">
<a href='https://app.cortexcpp.com/download/beta/windows-amd64-local'>
<img src='https://github.com/janhq/docs/blob/main/static/img/windows.png' style="height:14px; width: 14px" />
<b>cortex-local-installer.exe</b>
</a>
</td>
<td style="text-align:center">
<a href='https://app.cortexcpp.com/download/beta/windows-amd64-network'>
<img src='https://github.com/janhq/docs/blob/main/static/img/windows.png' style="height:14px; width: 14px" />
<b>cortex-network-installer.exe</b>
</a>
</td>
<td style="text-align:center">
<a href='https://app.cortexcpp.com/download/beta/mac-universal-local'>
<img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
<b>cortex-local-installer.pkg</b>
</a>
</td>
<td style="text-align:center">
<a href='https://app.cortexcpp.com/download/beta/mac-universal-network'>
<img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
<b>cortex-network-installer.pkg</b>
</a>
</td>
<td style="text-align:center">
<a href='https://app.cortexcpp.com/download/beta/linux-amd64-local'>
<img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:14px; width: 14px" />
<b>cortex-local-installer.deb</b>
</a>
</td>
<td style="text-align:center">
<a href='https://app.cortexcpp.com/download/beta/linux-amd64-network'>
<img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:14px; width: 14px" />
Expand All @@ -120,35 +225,17 @@ Cortex.cpp is a multi-engine that uses `llama.cpp` as the default engine but als
<tr style="text-align:center">
<td style="text-align:center"><b>Nightly Build (Experimental)</b></td>
<td style="text-align:center">
<a href='https://app.cortexcpp.com/download/nightly/windowns-amd64-local'>
<img src='https://github.com/janhq/docs/blob/main/static/img/windows.png' style="height:14px; width: 14px" />
<b>cortex-local-installer.exe</b>
</a>
</td>
<td style="text-align:center">
<a href='https://app.cortexcpp.com/download/nightly/windowns-amd64-network'>
<a href='https://app.cortexcpp.com/download/nightly/windows-amd64-network'>
<img src='https://github.com/janhq/docs/blob/main/static/img/windows.png' style="height:14px; width: 14px" />
<b>cortex-network-installer.exe</b>
</a>
</td>
<td style="text-align:center">
<a href='https://app.cortexcpp.com/download/nightly/mac-universal-local'>
<img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
<b>cortex-local-installer.pkg</b>
</a>
</td>
<td style="text-align:center">
<a href='https://app.cortexcpp.com/download/nightly/mac-universal-network'>
<img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
<b>cortex-network-installer.pkg</b>
</a>
</td>
<td style="text-align:center">
<a href='https://app.cortexcpp.com/download/nightly/linux-amd64-local'>
<img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:14px; width: 14px" />
<b>cortex-local-installer.deb</b>
</a>
</td>
<td style="text-align:center">
<a href='https://app.cortexcpp.com/download/nightly/linux-amd64-network'>
<img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:14px; width: 14px" />
Expand All @@ -158,89 +245,6 @@ Cortex.cpp is a multi-engine that uses `llama.cpp` as the default engine but als
</tr>
</table>

### Instructions

The local installer packages all required dependencies inside the installer itself, so you don’t need an internet connection during the installation process.
The network installer downloads the necessary dependencies from the internet during the installation. This option provides a smaller installer, but requires an internet connection.

After installation, you can run Cortex.cpp from the command line by typing `cortex --help`. For beta and nightly builds, you can run `cortex-beta --help` and `cortex-nightly --help` respectively.

#### Windows and MacOS

Download the installer and double-click to the exe file to start the installation process. Follow the on-screen instructions to complete the installation.

For MacOS, there is a uninstaller script comes with the binary and added to the `/usr/local/bin/` directory. The script is named `cortex-uninstall.sh` for stable builds, `cortex-beta-uninstall.sh` for beta builds and `cortex-nightly-uninstall.sh` for nightly builds.

#### Linux

Download the installer and run the following command in the terminal:

```bash
sudo apt install ./cortex-local-installer.deb
# or
sudo apt install ./cortex-network-installer.deb
```

The binary will be installed in the `/usr/bin/` directory.

## Built-in Model Library

Cortex.cpp supports various models available on the [Cortex Hub](https://huggingface.co/cortexso). Once downloaded, all model source files will be stored in `~\cortexcpp\models`.

Example models:

| Model | llama.cpp<br >`:gguf` | TensorRT<br >`:tensorrt` | ONNXRuntime<br >`:onnx` | Command |
|------------------|-----------------------|------------------------------------------|----------------------------|-------------------------------|
| llama3.1 | ✅ | | ✅ | cortex run llama3.1:gguf |
| llama3 | ✅ | ✅ | ✅ | cortex run llama3 |
| mistral | ✅ | ✅ | ✅ | cortex run mistral |
| qwen2 | ✅ | | | cortex run qwen2:7b-gguf |
| codestral | ✅ | | | cortex run codestral:22b-gguf |
| command-r | ✅ | | | cortex run command-r:35b-gguf |
| gemma | ✅ | | ✅ | cortex run gemma |
| mixtral | ✅ | | | cortex run mixtral:7x8b-gguf |
| openhermes-2.5 | ✅ | ✅ | ✅ | cortex run openhermes-2.5 |
| phi3 (medium) | ✅ | | ✅ | cortex run phi3:medium |
| phi3 (mini) | ✅ | | ✅ | cortex run phi3:mini |
| tinyllama | ✅ | | | cortex run tinyllama:1b-gguf |

> **Note**:
> You should have at least 8 GB of RAM available to run the 7B models, 16 GB to run the 14B models, and 32 GB to run the 32B models.

## Cortex.cpp CLI Commands
For complete details on CLI commands, please refer to our [CLI documentation](https://cortex.so/docs/cli).

## REST API
Cortex.cpp includes a REST API accessible at `localhost:3928`. For a complete list of endpoints and their usage, visit our [API documentation](https://cortex.so/api-reference).

## Uninstallation

### Windows
1. Open the Windows Control Panel.
2. Navigate to `Add or Remove Programs`.
3. Search for `cortexcpp` and double click to uninstall. (for beta and nightly builds, search for `cortexcpp-beta` and `cortexcpp-nightly` respectively)

### MacOs
Run the uninstaller script:
```bash
# For stable builds
sudo sh cortex-uninstall.sh
# For beta builds
sudo sh cortex-beta-uninstall.sh
# For nightly builds
sudo sh cortex-nightly-uninstall.sh
```

### Linux
```bash
# For stable builds
sudo apt remove cortexcpp
# For beta builds
sudo apt remove cortexcpp-beta
# For nightly builds
sudo apt remove cortexcpp-nightly
```

### Build from Source

#### Windows
Expand Down Expand Up @@ -318,9 +322,26 @@ make -j4
cortex
```

## Uninstallation
### Windows
1. Open the Windows Control Panel.
2. Navigate to `Add or Remove Programs`.
3. Search for `cortexcpp` and double click to uninstall. (for beta and nightly builds, search for `cortexcpp-beta` and `cortexcpp-nightly` respectively)

### MacOs
Run the uninstaller script:
```bash
sudo sh cortex-uninstall.sh
```
For MacOS, there is a uninstaller script comes with the binary and added to the `/usr/local/bin/` directory. The script is named `cortex-uninstall.sh` for stable builds, `cortex-beta-uninstall.sh` for beta builds and `cortex-nightly-uninstall.sh` for nightly builds.

### Linux
```bash
# For stable builds
sudo apt remove cortexcpp
```

## Contact Support
- For support, please file a [GitHub ticket](https://github.com/janhq/cortex/issues/new/choose).
- For support, please file a [GitHub ticket](https://github.com/janhq/cortex.cpp/issues/new/choose).
- For questions, join our Discord [here](https://discord.gg/FTk2MvZwJH).
- For long-form inquiries, please email [[email protected]](mailto:[email protected]).