Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: remove outdated links #1319

Merged
merged 3 commits into from
Sep 26, 2024
Merged

docs: remove outdated links #1319

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
43cf5a4
docs: remove outdated links
Sep 24, 2024
e607fbd
fix: header
Sep 24, 2024
5bb6dc9
Merge branch 'dev' into updateReadme
freelerobot Sep 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
221 changes: 9 additions & 212 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
- <a href="https://github.com/janhq/cortex.cpp/releases">Changelog</a> - <a href="https://github.com/janhq/cortex.cpp/issues">Bug reports</a> - <a href="https://discord.gg/AsJ8krTT3N">Discord</a>
</p>

> ⚠️ **Cortex.cpp is currently in Development. This documentation outlines the intended behavior of Cortex, which may not yet be fully implemented in the codebase.**
> ⚠️ **Cortex.cpp is currently in active development. This outlines the intended behavior of Cortex, which may not yet be fully implemented in the codebase.**

## Overview
Cortex.cpp is a Local AI engine that is used to run and customize LLMs. Cortex can be deployed as a standalone server, or integrated into apps like [Jan.ai](https://jan.ai/).
Expand All @@ -28,142 +28,15 @@ Cortex.cpp is a multi-engine that uses `llama.cpp` as the default engine but als
- [`onnx`](https://github.com/janhq/cortex.onnx)
- [`tensorrt-llm`](https://github.com/janhq/cortex.tensorrt-llm)

To install Cortex.cpp, download the installer for your operating system from the following options:

<table>
<tr style="text-align:center">
<td style="text-align:center"><b>Version Type</b></td>
<td style="text-align:center"><b>Windows</b></td>
<td colspan="2" style="text-align:center"><b>MacOS</b></td>
<td colspan="2" style="text-align:center"><b>Linux</b></td>
</tr>
<tr style="text-align:center">
<td style="text-align:center"><b>Stable (Recommended)</b></td>
<td style="text-align:center">
<a href='https://github.com/janhq/cortex.cpp/releases'>
<img src='https://github.com/janhq/docs/blob/main/static/img/windows.png' style="height:14px; width: 14px" />
<b>Download</b>
</a>
</td>
<td style="text-align:center">
<a href='https://github.com/janhq/cortex.cpp/releases'>
<img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
<b>Intel</b>
</a>
</td>
<td style="text-align:center">
<a href='https://github.com/janhq/cortex.cpp/releases'>
<img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
<b>M1/M2/M3/M4</b>
</a>
</td>
<td style="text-align:center">
<a href='https://github.com/janhq/cortex.cpp/releases'>
<img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:14px; width: 14px" />
<b>Debian Download</b>
</a>
</td>
<td style="text-align:center">
<a href='https://github.com/janhq/cortex.cpp/releases'>
<img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:14px; width: 14px" />
<b>Fedora Download</b>
</a>
</td>
</tr>
</table>
## Installation

> **Note**:
> You can also build Cortex.cpp from source by following the steps [here](#build-from-source).


### Libraries
- [cortex.js](https://github.com/janhq/cortex.js)
- [cortex.py](https://github.com/janhq/cortex-python)

## Quickstart
### CLI
```bash
# 1. Start the Cortex.cpp server (The server is running at localhost:3928)
cortex start

# 2. Start and chat with the model
cortex run <model_id>:[engine_name]

# 3. Stop a model
cortex stop <model_id>:[engine_name]

# 4. Stop the Cortex.cpp server
cortex stop
```
### API
1. Start the API server using `cortex` command.
2. **Pull a Model**
```bash
curl --request POST \
--url http://localhost:3928/v1/models/{model_id}/pull
```

3. **Start a Model**
```bash
curl --request POST \
--url http://localhost:3928/v1/models/{model_id}/start \
--header 'Content-Type: application/json' \
--data '{
"prompt_template": "system\n{system_message}\nuser\n{prompt}\nassistant",
"stop": [],
"ngl": 4096,
"ctx_len": 4096,
"cpu_threads": 10,
"n_batch": 2048,
"caching_enabled": true,
"grp_attn_n": 1,
"grp_attn_w": 512,
"mlock": false,
"flash_attn": true,
"cache_type": "f16",
"use_mmap": true,
"engine": "llamacpp"
}'
```

4. **Chat with a Model**
```bash
curl http://localhost:3928/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "",
"messages": [
{
"role": "user",
"content": "Hello"
},
],
"model": "mistral",
"stream": true,
"max_tokens": 1,
"stop": [
null
],
"frequency_penalty": 1,
"presence_penalty": 1,
"temperature": 1,
"top_p": 1
}'
```

5. **Stop a Model**
```bash
curl --request POST \
--url http://localhost:3928/v1/models/mistral/stop
```
6. Stop the Cortex.cpp server using `cortex stop` command.
> **Note**:
> Our API server is fully compatible with the OpenAI API, making it easy to integrate with any systems or tools that support OpenAI-compatible APIs.
You can install a nightly (unstable) version of Cortex from Discord here: https://discord.gg/nGp6PMrUqS

## Built-in Model Library
Cortex.cpp supports various models available on the [Cortex Hub](https://huggingface.co/cortexso). Once downloaded, all model source files will be stored at `C:\Users\<username>\AppData\Local\cortexcpp\models`.

Here are example of models that you can use based on each supported engine:
Cortex.cpp supports various models available on the [Cortex Hub](https://huggingface.co/cortexso). Once downloaded, all model source files will be stored in `~\cortexcpp\models`.

Example models:

| Model | llama.cpp<br >`:gguf` | TensorRT<br >`:tensorrt` | ONNXRuntime<br >`:onnx` | Command |
|------------------|-----------------------|------------------------------------------|----------------------------|-------------------------------|
Expand All @@ -190,6 +63,7 @@ For complete details on CLI commands, please refer to our [CLI documentation](ht
Cortex.cpp includes a REST API accessible at `localhost:3928`. For a complete list of endpoints and their usage, visit our [API documentation](https://cortex.so/api-reference).

## Uninstallation

### Windows
1. Navigate to `Add or Remove Programs`.
2. Search for Cortex.cpp and click `Uninstall`.
Expand All @@ -205,83 +79,6 @@ sudo sh cortex-uninstall.sh
sudo apt remove cortexcpp
```

## Alternate Installation
We also provide Beta and Nightly version.
<table>
<tr style="text-align:center">
<td style="text-align:center"><b>Version Type</b></td>
<td style="text-align:center"><b>Windows</b></td>
<td colspan="2" style="text-align:center"><b>MacOS</b></td>
<td colspan="2" style="text-align:center"><b>Linux</b></td>
</tr>
<tr style="text-align:center">
<td style="text-align:center"><b>Beta Build</b></td>
<td style="text-align:center">
<a href='https://github.com/janhq/cortex.cpp/releases'>
<img src='https://github.com/janhq/docs/blob/main/static/img/windows.png' style="height:14px; width: 14px" />
<b>cortexcpp.exe</b>
</a>
</td>
<td style="text-align:center">
<a href='https://github.com/janhq/cortex.cpp/releases'>
<img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
<b>Intel</b>
</a>
</td>
<td style="text-align:center">
<a href='https://github.com/janhq/cortex.cpp/releases'>
<img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
<b>M1/M2/M3/M4</b>
</a>
</td>
<td style="text-align:center">
<a href='https://github.com/janhq/cortex.cpp/releases'>
<img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:14px; width: 14px" />
<b>cortexcpp.deb</b>
</a>
</td>
<td style="text-align:center">
<a href='https://github.com/janhq/cortex.cpp/releases'>
<img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:14px; width: 14px" />
<b>cortexcpp.AppImage</b>
</a>
</td>
</tr>
<tr style="text-align:center">
<td style="text-align:center"><b>Nightly Build</b></td>
<td style="text-align:center">
<a href='https://github.com/janhq/cortex.cpp/releases'>
<img src='https://github.com/janhq/docs/blob/main/static/img/windows.png' style="height:14px; width: 14px" />
<b>cortexcpp.exe</b>
</a>
</td>
<td style="text-align:center">
<a href='https://github.com/janhq/cortex.cpp/releases'>
<img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
<b>Intel</b>
</a>
</td>
<td style="text-align:center">
<a href='https://github.com/janhq/cortex.cpp/releases'>
<img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
<b>M1/M2/M3/M4</b>
</a>
</td>
<td style="text-align:center">
<a href='https://github.com/janhq/cortex.cpp/releases'>
<img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:14px; width: 14px" />
<b>cortexcpp.deb</b>
</a>
</td>
<td style="text-align:center">
<a href='https://github.com/janhq/cortex.cpp/releases'>
<img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:14px; width: 14px" />
<b>cortexcpp.AppImage</b>
</a>
</td>
</tr>
</table>

### Build from Source

#### Windows
Expand Down Expand Up @@ -355,8 +152,8 @@ make -j4
6. Verify that Cortex.cpp is installed correctly by getting help information.

```sh
# Get the help information
cortex -h
# Get help
cortex
```

## Contact Support
Expand Down