Cortex.cpp: Installation & Uninstallation

[namchuai]

Goal

• Installer can be updated rapidly (vs. waiting for Package Manager approvals)
• Installer supports Stable, Beta and Nightly versions

Major Decisions

• We will have native installers, as package manager approvals take time
• We will install llama.cpp engine by default, at install time (i.e. cortex engines init llamacpp)
• Use native installers to take advantage of pre-install and post-install hooks (.exe, .dmg, .deb and .appimage)

Out-of-scope (for now)

• "Full Installer" for air-gapped use cases (i.e. unable to pull llama.cpp at install-time)

Related Issues

• feat: Jan has a portable / non install version jan#2643
• Discussion: Cortex.cpp Uninstallation #775
• feat: Support Steam Deck jan#3360
• bug: users cannot do a fresh cortex install due to dangling cortex folder (from previous install) #1082

Cortex's Installed Files

Installer:

• Windows
  • Format: .exe
  • Binary location:
    • Stable: C:\Users\<username>\AppData\Local\cortexcpp\cortex.exe
    • Beta: C:\Users\<username>\AppData\Local\cortexcpp-beta\cortex-beta.exe
    • Nighty: C:\Users\<username>\AppData\Local\cortexcpp-nightly\cortex-nightly.exe
  • Configuration file:
    • Stable: C:\Users\<username>\.cortexrc
    • Beta: C:\Users\<username>\.cortexrc-beta
    • Nighty: C:\Users\<username>\.cortexrc-nightly
  • Data folder (contains engines, models, logs)
    • Stable: C:\Users\<username>\.cortexcpp
    • Beta: C:\Users\<username>\.cortexcpp-beta
    • Nighty: C:\Users\<username>\.cortexcpp-nightly
• Linux:
  • Format: .deb
  • Binary location:
    • Stable: /usr/bin/cortexcpp
    • Beta: /usr/bin/cortexcpp-beta
    • Nighty: /usr/bin/cortexcpp-nightly
  • Configuration file:
    • Stable: /home/<username>/.cortexrc
    • Beta: /home/<username>/.cortexrc-beta
    • Nighty: /home/<username>/.cortexrc-nightly
  • Data folder (contains engines, models, logs)
    • Stable: /home/<username>/.cortexcpp
    • Beta: /home/<username>/.cortexcpp-beta
    • Nighty: /home/<username>/.cortexcpp-nightly
• Macos:
  • Format: .pkg
  • Binary location:
    • Stable: /usr/local/bin/cortexcpp
    • Beta: /usr/local/bin/cortexcpp-beta
    • Nighty: /usr/local/bin/cortexcpp-nightly
  • Configuration file:
    • Stable: /Users/<username>/.cortexrc
    • Beta: /Users/<username>/.cortexrc-beta
    • Nighty: /Users/<username>/.cortexrc-nightly
  • Data folder (contains engines, models, logs)
    • Stable: /Users/<username>/.cortexcpp
    • Beta: /Users/<username>/.cortexcpp-beta
    • Nighty: /Users/<username>/.cortexcpp-nightly

Original Post

Motivation
Couple of options:

1. via Package managers (brew, winget, launchpad, etc.)

• Pros:
  • Easy to update (maybe?) just provide package manager team
  • Easy for acquire user? (like just running brew install cortexcpp or something like that)
• Cons:
  • Take long time for new version come out
  • Hard (or impossible) to choose a place to install binary/files (not entirely sure. Open for discussion)

2. self hosted

• Pros:
  • Not being constrained by Package managers
  • Shorter time to market
  • Easy to have multiple variants (stable, beta, nightly)
  • Let user decide where to store files. More managable?
• Cons:
  • Have to handle upgrade ourselves
  • User acquisition is more difficult?

3. Hmm, both?

[hiento09]

Installer:

• Windows
  • Format: .exe
  • Binary location:
    • Stable: C:\Users\<username>\AppData\Local\cortex\cortex.exe
    • Beta: C:\Users\<username>\AppData\Local\cortex-beta\cortex-beta.exe
    • Nighty: C:\Users\<username>\AppData\Local\cortex-nightly\cortex-nightly.exe
  • Configuration file:
    • Stable: C:\Users\<username>\.cortexrc
    • Beta: C:\Users\<username>\.cortexrc-beta
    • Nighty: C:\Users\<username>\.cortexrc-nightly
  • Data folder (contains engines, models, logs)
    • Stable: C:\Users\<username>\.cortexcpp
    • Beta: C:\Users\<username>\.cortexcpp-beta
    • Nighty: C:\Users\<username>\.cortexcpp-nightly
• Linux:
  • Format: .deb
  • Binary location:
    • Stable: /usr/bin/cortex
    • Beta: /usr/bin/cortex-beta
    • Nighty: /usr/bin/cortex-nightly
  • Configuration file:
    • Stable: /home/<username>/.cortexrc
    • Beta: /home/<username>/.cortexrc-beta
    • Nighty: /home/<username>/.cortexrc-nightly
  • Data folder (contains engines, models, logs)
    • Stable: /home/<username>/.cortex
    • Beta: /home/<username>/.cortexcpp-beta
    • Nighty: /home/<username>/.cortexcpp-nightly
• Macos:
  • Format: .pkg
  • Binary location:
    • Stable: /usr/local/bin/cortex
    • Beta: /usr/local/bin/cortex-beta
    • Nighty: /usr/local/bin/cortex-nightly
  • Configuration file:
    • Stable: /Users/<username>/.cortexrc
    • Beta: /Users/<username>/.cortexrc-beta
    • Nighty: /Users/<username>/.cortexrc-nightly
  • Data folder (contains engines, models, logs)
    • Stable: /Users/<username>/.cortexcpp
    • Beta: /Users/<username>/.cortexcpp-beta
    • Nighty: /Users/<username>/.cortexcpp-nightly

[freelerobot]

@hiento09 I spot some inconsistencies in your folder/file naming vs #1040

[hiento09]

@hiento09 I spot some inconsistencies in your folder/file naming vs #1040

I just edited my comment base on this comment https://github.com/janhq/cortex/issues/1040#issuecomment-2327938276 from @dan-homebrew

[dan-menlo]

@hiento09 I made some changes to your folder/binary layout, based on our earlier discussion #1092

Changes

• CLI command is cortex, and binary name is cortex (or cortex-beta, cortex-nightly)
• Otherwise, our product is cortexcpp (with versions: cortexcpp-beta, cortexcpp-nightly)
• Our convention is to hyphenate product-version
• Data folder is pending discussion on visible vs. hidden folder in Discussion: Cortex.cpp Data Structures #1040

Installer:

• Windows
  • Format: .exe
  • Binary location:
    • Stable: C:\Users\<username>\AppData\Local\cortexcpp\cortex.exe
    • Beta: C:\Users\<username>\AppData\Local\cortexcpp-beta\cortex-beta.exe
    • Nighty: C:\Users\<username>\AppData\Local\cortexcpp-nightly\cortex-nightly.exe
  • Configuration file:
    • Stable: C:\Users\<username>\.cortexrc
    • Beta: C:\Users\<username>\.cortexrc-beta
    • Nighty: C:\Users\<username>\.cortexrc-nightly
  • Data folder (contains engines, models, logs)
    • Stable: C:\Users\<username>\.cortexcpp
    • Beta: C:\Users\<username>\.cortexcpp-beta
    • Nighty: C:\Users\<username>\.cortexcpp-nightly
• Linux:
  • Format: .deb
  • Binary location:
    • Stable: /usr/bin/cortexcpp
    • Beta: /usr/bin/cortexcpp-beta
    • Nighty: /usr/bin/cortexcpp-nightly
  • Configuration file:
    • Stable: /home/<username>/.cortexrc
    • Beta: /home/<username>/.cortexrc-beta
    • Nighty: /home/<username>/.cortexrc-nightly
  • Data folder (contains engines, models, logs)
    • Stable: /home/<username>/.cortexcpp
    • Beta: /home/<username>/.cortexcpp-beta
    • Nighty: /home/<username>/.cortexcpp-nightly
• Macos:
  • Format: .pkg
  • Binary location:
    • Stable: /usr/local/bin/cortexcpp
    • Beta: /usr/local/bin/cortexcpp-beta
    • Nighty: /usr/local/bin/cortexcpp-nightly
  • Configuration file:
    • Stable: /Users/<username>/.cortexrc
    • Beta: /Users/<username>/.cortexrc-beta
    • Nighty: /Users/<username>/.cortexrc-nightly
  • Data folder (contains engines, models, logs)
    • Stable: /Users/<username>/.cortexcpp
    • Beta: /Users/<username>/.cortexcpp-beta
    • Nighty: /Users/<username>/.cortexcpp-nightly

[hiento09]

@hiento09 I made some changes to your folder/binary layout, based on our earlier discussion #1092

Changes

• CLI command is cortex, and binary name is cortex (or cortex-beta, cortex-nightly)

• Our convention is to hyphenate name-version

• Data folder is pending discussion on visible vs. hidden folder in Discussion: Cortex.cpp Data Structures #1040

Installer:

• Windows

  • Format: .exe

  • Binary location:

    • Stable: C:\Users\<username>\AppData\Local\cortexcpp\cortex.exe

    • Beta: C:\Users\<username>\AppData\Local\cortexcpp-beta\cortex-beta.exe

    • Nighty: C:\Users\<username>\AppData\Local\cortexcpp-nightly\cortex-nightly.exe

  • Configuration file:

    • Stable: C:\Users\<username>\.cortexrc

    • Beta: C:\Users\<username>\.cortexrc-beta

    • Nighty: C:\Users\<username>\.cortexrc-nightly

  • Data folder (contains engines, models, logs)

    • Stable: C:\Users\<username>\.cortexcpp

    • Beta: C:\Users\<username>\.cortexcpp-beta

    • Nighty: C:\Users\<username>\.cortexcpp-nightly

• Linux:

  • Format: .deb

  • Binary location:

    • Stable: /usr/bin/cortexcpp

    • Beta: /usr/bin/cortexcpp-beta

    • Nighty: /usr/bin/cortexcpp-nightly

  • Configuration file:

    • Stable: /home/<username>/.cortexrc

    • Beta: /home/<username>/.cortexrc-beta

    • Nighty: /home/<username>/.cortexrc-nightly

  • Data folder (contains engines, models, logs)

    • Stable: /home/<username>/.cortexcpp

    • Beta: /home/<username>/.cortexcpp-beta

    • Nighty: /home/<username>/.cortexcpp-nightly

• Macos:

  • Format: .pkg

  • Binary location:

    • Stable: /usr/local/bin/cortexcpp

    • Beta: /usr/local/bin/cortexcpp-beta

    • Nighty: /usr/local/bin/cortexcpp-nightly

  • Configuration file:

    • Stable: /Users/<username>/.cortexrc

    • Beta: /Users/<username>/.cortexrc-beta

    • Nighty: /Users/<username>/.cortexrc-nightly

  • Data folder (contains engines, models, logs)

    • Stable: /Users/<username>/.cortexcpp

    • Beta: /Users/<username>/.cortexcpp-beta

    • Nighty: /Users/<username>/.cortexcpp-nightly

Thank you @dan-homebrew

[hiento09]

Task to do here #1030

[hiento09]

For installer, llamacpp engine will be installed by default

[dan-menlo]

@hiento09 Can I clarify my understanding of the Installer, as there are a couple of things that are not aligned from our discussion last week:

Last week's discussion: https://link.excalidraw.com/l/kFY0dI05mm/3rpQK5NI8Xb

Installer Contents

• Cortex will ship with llama.cpp as a default engine
• TensorRT-LLM and ONNX will (for now) be initialized by running cortex init <engine>

Installation Methods

For the 3 operating systems, my understanding was as follows:

• Windows: .exe
• Mac: .dmg
• Linux: Install script (we would handle .deb and .appimage later on), in the format of:

curl -fsSL https://get.cortex.so -o get-cortex.sh

From the post above, it seems like we're going directly to .deb and .appImage.

My question: Does the .deb and .appImage wrap the Installer Script? There are definite advantages to having a .deb and .appimage, I just want to know how much more effort it is.

Installation Logic

My understanding is that our Installation logic focuses on two tasks:

• Hardware Detection (e.g. GPU type, and CPU instruction set - e.g. AVX2, AVX512)
• Pulling the correct version of llama.cpp at install time

Alternatives

I am increasingly thinking that it is a better idea for us to pack a larger installer:

• Cortex will be used in offline environments (e.g. USB install), and packing a full installer is better
• We can delete the unused llama.cpp versions during the installation process, to optimize long-term disk usage
• This seems to be the approach taken by the other llama.cpp UIs and libraries (e.g. Ollama, LMStudio)

[hiento09]

My question: Does the .deb and .appImage wrap the Installer Script? There are definite advantages to having a .deb and .appimage, I just want to know how much more effort it is.

This is an example of template structure for .deb file in linux

Yes, it wraps bash scripts and allows us to interact with hooks like pre-install, post-install, pre-uninstall, and post-uninstall. It also has installation and uninstallation functions. If we go with a bash script, then we have to write all of the above functions ourselves.

[dan-menlo]

@hiento09 Ok, let's go with .deb and .appimage then.

[hiento09]

Link the final decision for naming here #1092

[freelerobot]

@hiento09 one of our macOS users found Jan files in
~/Library/Preferences/ByHost/jan.ai.app.ShipIt.<UUID>.plist
~/Library/Preferences/jan.ai.app.plist

1. Is this something that is automatically cleared by OS when we uninstall on mac
2. Will we need to take care of this dangling file.
3. File naming wise, is it supposed to be inverse , i.e. ai.jan.app...

For 3: see examples:

com.microsoft.VSCode.ShipIt.UUID.plist
org.whispersystems.signal-desktop.ShipIt.UUID.plist

[hiento09]

@0xSage , cc @louis-jan

1. Is this something that is automatically cleared by OS when we uninstall on mac => Yes
2. Will we need to take care of this dangling file. => No, it's a part of Jan app and is taking care by electron-builder
3. File naming wise, is it supposed to be inverse , i.e. ai.jan.app... => Yes, we can change to whatever name, just notify that change app ID will cause auto-updater crash

[freelerobot]

then keep the naming - no need to break things unnecessarily 🙏

…

On Tue, Sep 10, 2024 at 12:23 PM hiento09 ***@***.***> wrote: @0xSage <https://github.com/0xSage> , cc @louis-jan <https://github.com/louis-jan> 1. Is this something that is automatically cleared by OS when we uninstall on mac => Yes 2. Will we need to take care of this dangling file. => No, it's a part of Jan app and is taking care by electron-builder 3. File naming wise, is it supposed to be inverse , i.e. ai.jan.app... => Yes, we can change to whatever name, just notify that change app ID will cause auto-updater crash — Reply to this email directly, view it on GitHub <#1114 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQVWFCF3LZG6LLZUCNP2AFDZVZX2TAVCNFSM6AAAAABNWZWFBKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANJZG44DGMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>