epic: Separate CLI and API processes

[gabrielle-ong]

Goal

• Cortex will primarily be an OpenAI API equivalent server
• Cortex CLI should be a layer on top of the API Server
• CLI and server are now two distinct binaries

Subtasks

• planning: Cortex CLI should be a layer on top of the API Server #1492
• epic: installer and uninstaller for cortex cli and cortex api binaries #1480

Discussion:

#1386

Success Criteria (windows, max, linux)

• Installer: Installer installs 2 different binaries
• Uninstaller: Uninstalls 2 different binaries
• Cortex update replaces the app, installer, uninstaller and binary file (without installing cortex.llamacpp)
• all CLI commands will start API server, if not running
• Edge case: User run cortex update on v1.0.0 → prompted to install cortex-server as with 1.0.1

[hiento09]

Hey guys @dan-homebrew @vansangpfiev @gabrielle-ong @namchuai @nguyenhoangthuan99 @0xSage , I’d like to present an idea for the installer and updater when we separate the CLI and server into two distinct binaries, as follows:

Given that:

In the current updater, the user runs the cortex update command, which causes the cortex process to download the new binary version and replace the existing binary. However, other components of the installer, such as the uninstaller and post-uninstall script, are not updated because we only replace the binary file, not the entire app version on the user's machine.

Solution:

Instead of the current cortex update only downloading and replacing the binary file, the cortex update command will pull the installer file and install it in headless mode using the following commands:

• macos: touch /var/tmp/cortex_installer_skip_postinstall && sudo installer -pkg cortexcpp-nightly-network.pkg -target / && rm /var/tmp/cortex_installer_skip_postinstall
• Linux: echo -e "n\n" | sudo SKIP_POSTINSTALL=true apt install -y --allow-downgrades ./cortexcpp-nightly-network.deb
• Windows:
  • Powershell Start-Process -FilePath ".\cortex-0.5.0-154-windows-amd64-network-installer.exe" -ArgumentList "/VERYSILENT /SUPPRESSMSGBOXES /NORESTART /SkipPostInstall" -Wait
  • CMD start /wait "" network-setup.exe /VERYSILENT /SUPPRESSMSGBOXES /NORESTART /SkipPostInstall
    In these commands, we skip the postinstall process, which installs the cortex.llamacpp engine. This allows the installer to replace only the app, installer, uninstaller, and the new binary file. With this approach, the Cortex code no longer needs to handle the complexity of replacing two binaries; it can simply run the commands above to update both binaries and the scripts in the installer and uninstaller simultaneously.

What do you guys think of this idea? Please leave your comments. Thanks!

[vansangpfiev]

Hey guys @dan-homebrew @vansangpfiev @gabrielle-ong @namchuai @nguyenhoangthuan99 @0xSage , I’d like to present an idea for the installer and updater when we separate the CLI and server into two distinct binaries, as follows:

Given that:

In the current updater, the user runs the cortex update command, which causes the cortex process to download the new binary version and replace the existing binary. However, other components of the installer, such as the uninstaller and post-uninstall script, are not updated because we only replace the binary file, not the entire app version on the user's machine.

Solution:

Instead of the current cortex update only downloading and replacing the binary file, the cortex update command will pull the installer file and install it in headless mode using the following commands:

• macos: SKIP_POSTINSTALL=true && sudo installer -pkg cortex-1.0.0-rc1-mac-universal-network-installer.pkg -target /
• Linux: echo -e "n\n" | SKIP_POSTINSTALL=true && sudo apt install -y --allow-downgrades ./cortex-1.0.0-165-linux-amd64-network-installer.deb
• Windows: Start-Process -FilePath ".\cortex-0.5.0-154-windows-amd64-network-installer.exe" -ArgumentList "/SkipPostInstall /VERYSILENT /SUPPRESSMSGBOXES /NORESTART" -Wait

In these commands, we skip the postinstall process, which installs the cortex.llamacpp engine. This allows the installer to replace only the app, installer, uninstaller, and the new binary file. With this approach, the Cortex code no longer needs to handle the complexity of replacing two binaries; it can simply run the commands above to update both binaries and the scripts in the installer and uninstaller simultaneously.

What do you guys think of this idea? Please leave your comments. Thanks!

I totally agree with the above solution. That way, we won't need to change much of the updater code in the long-term.
While calling an installer from cortex can be tricky, it is doable.

[freelerobot]

Largely sounds good. From the user's perspective, what is the experience like? - Do new windows/views pop up (if so, then no go) - Is there an increased update wait time?

…

On Tue, Oct 15, 2024 at 7:35 AM vansangpfiev ***@***.***> wrote: Hey guys @dan-homebrew <https://github.com/dan-homebrew> @vansangpfiev <https://github.com/vansangpfiev> @gabrielle-ong <https://github.com/gabrielle-ong> @namchuai <https://github.com/namchuai> @nguyenhoangthuan99 <https://github.com/nguyenhoangthuan99> @0xSage <https://github.com/0xSage> , I’d like to present an idea for the installer and updater when we separate the CLI and server into two distinct binaries, as follows: Given that: In the current updater, the user runs the cortex update command, which causes the cortex process to download the new binary version and replace the existing binary. However, other components of the installer, such as the uninstaller and post-uninstall script, are not updated because we only replace the binary file, not the entire app version on the user's machine. Solution: Instead of the current cortex update only downloading and replacing the binary file, the cortex update command will pull the installer file and install it in headless mode using the following commands: - macos: SKIP_POSTINSTALL=true && sudo installer -pkg cortex-1.0.0-rc1-mac-universal-network-installer.pkg -target / - Linux: echo -e "n\n" | SKIP_POSTINSTALL=true && sudo apt install -y --allow-downgrades ./cortex-1.0.0-165-linux-amd64-network-installer.deb - Windows: Start-Process -FilePath ".\cortex-0.5.0-154-windows-amd64-network-installer.exe" -ArgumentList "/SkipPostInstall /VERYSILENT /SUPPRESSMSGBOXES /NORESTART" -Wait In these commands, we skip the postinstall process, which installs the cortex.llamacpp engine. This allows the installer to replace only the app, installer, uninstaller, and the new binary file. With this approach, the Cortex code no longer needs to handle the complexity of replacing two binaries; it can simply run the commands above to update both binaries and the scripts in the installer and uninstaller simultaneously. What do you guys think of this idea? Please leave your comments. Thanks! I totally agree with the above solution. That way, we won't need to change much of the updater code in the long-term. While calling an installer from cortex can be tricky, it is doable. — Reply to this email directly, view it on GitHub <#1415 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQVWFCCBRX42LWHW2W7JDYDZ3RIL7AVCNFSM6AAAAABPJ5KC3WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJSGUZDCMZRGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[hiento09]

Largely sounds good. From the user's perspective, what is the experience like? - Do new windows/views pop up (if so, then no go) - Is there an increased update wait time?
…
On Tue, Oct 15, 2024 at 7:35 AM vansangpfiev @.> wrote: Hey guys @dan-homebrew https://github.com/dan-homebrew @vansangpfiev https://github.com/vansangpfiev @gabrielle-ong https://github.com/gabrielle-ong @namchuai https://github.com/namchuai @nguyenhoangthuan99 https://github.com/nguyenhoangthuan99 @0xSage https://github.com/0xSage , I’d like to present an idea for the installer and updater when we separate the CLI and server into two distinct binaries, as follows: Given that: In the current updater, the user runs the cortex update command, which causes the cortex process to download the new binary version and replace the existing binary. However, other components of the installer, such as the uninstaller and post-uninstall script, are not updated because we only replace the binary file, not the entire app version on the user's machine. Solution: Instead of the current cortex update only downloading and replacing the binary file, the cortex update command will pull the installer file and install it in headless mode using the following commands: - macos: SKIP_POSTINSTALL=true && sudo installer -pkg cortex-1.0.0-rc1-mac-universal-network-installer.pkg -target / - Linux: echo -e "n\n" | SKIP_POSTINSTALL=true && sudo apt install -y --allow-downgrades ./cortex-1.0.0-165-linux-amd64-network-installer.deb - Windows: Start-Process -FilePath ".\cortex-0.5.0-154-windows-amd64-network-installer.exe" -ArgumentList "/SkipPostInstall /VERYSILENT /SUPPRESSMSGBOXES /NORESTART" -Wait In these commands, we skip the postinstall process, which installs the cortex.llamacpp engine. This allows the installer to replace only the app, installer, uninstaller, and the new binary file. With this approach, the Cortex code no longer needs to handle the complexity of replacing two binaries; it can simply run the commands above to update both binaries and the scripts in the installer and uninstaller simultaneously. What do you guys think of this idea? Please leave your comments. Thanks! I totally agree with the above solution. That way, we won't need to change much of the updater code in the long-term. While calling an installer from cortex can be tricky, it is doable. — Reply to this email directly, view it on GitHub <#1415 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQVWFCCBRX42LWHW2W7JDYDZ3RIL7AVCNFSM6AAAAABPJ5KC3WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJSGUZDCMZRGM . You are receiving this because you were mentioned.Message ID: @.>

@0xSage

• Do new windows/views pop up (if so, then no go)? => No, all the commands above run the installer in headless mode. I tested them, and they work as expected.
• Is there an increased update wait time? => We use a network installation but disable the post-install script that pulls cortex.llamacpp from the internet. Instead, it only pulls the new binaries, which are less than 30MB. I tested this, and the update process is smooth and quick.

[freelerobot]

👍👍

…

On Tue, Oct 15, 2024 at 9:53 AM hiento09 ***@***.***> wrote: Largely sounds good. From the user's perspective, what is the experience like? - Do new windows/views pop up (if so, then no go) - Is there an increased update wait time? … <#m_2046471413798012038_> On Tue, Oct 15, 2024 at 7:35 AM vansangpfiev *@*.*> wrote: Hey guys @dan-homebrew <https://github.com/dan-homebrew> https://github.com/dan-homebrew <https://github.com/dan-homebrew> @vansangpfiev <https://github.com/vansangpfiev> https://github.com/vansangpfiev <https://github.com/vansangpfiev> @gabrielle-ong <https://github.com/gabrielle-ong> https://github.com/gabrielle-ong <https://github.com/gabrielle-ong> @namchuai <https://github.com/namchuai> https://github.com/namchuai <https://github.com/namchuai> @nguyenhoangthuan99 <https://github.com/nguyenhoangthuan99> https://github.com/nguyenhoangthuan99 <https://github.com/nguyenhoangthuan99> @0xSage <https://github.com/0xSage> https://github.com/0xSage <https://github.com/0xSage> , I’d like to present an idea for the installer and updater when we separate the CLI and server into two distinct binaries, as follows: Given that: In the current updater, the user runs the cortex update command, which causes the cortex process to download the new binary version and replace the existing binary. However, other components of the installer, such as the uninstaller and post-uninstall script, are not updated because we only replace the binary file, not the entire app version on the user's machine. Solution: Instead of the current cortex update only downloading and replacing the binary file, the cortex update command will pull the installer file and install it in headless mode using the following commands: - macos: SKIP_POSTINSTALL=true && sudo installer -pkg cortex-1.0.0-rc1-mac-universal-network-installer.pkg -target / - Linux: echo -e "n\n" | SKIP_POSTINSTALL=true && sudo apt install -y --allow-downgrades ./cortex-1.0.0-165-linux-amd64-network-installer.deb - Windows: Start-Process -FilePath ".\cortex-0.5.0-154-windows-amd64-network-installer.exe" -ArgumentList "/SkipPostInstall /VERYSILENT /SUPPRESSMSGBOXES /NORESTART" -Wait In these commands, we skip the postinstall process, which installs the cortex.llamacpp engine. This allows the installer to replace only the app, installer, uninstaller, and the new binary file. With this approach, the Cortex code no longer needs to handle the complexity of replacing two binaries; it can simply run the commands above to update both binaries and the scripts in the installer and uninstaller simultaneously. What do you guys think of this idea? Please leave your comments. Thanks! I totally agree with the above solution. That way, we won't need to change much of the updater code in the long-term. While calling an installer from cortex can be tricky, it is doable. — Reply to this email directly, view it on GitHub <#1415 (comment) <#1415 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQVWFCCBRX42LWHW2W7JDYDZ3RIL7AVCNFSM6AAAAABPJ5KC3WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJSGUZDCMZRGM <https://github.com/notifications/unsubscribe-auth/AQVWFCCBRX42LWHW2W7JDYDZ3RIL7AVCNFSM6AAAAABPJ5KC3WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJSGUZDCMZRGM> . You are receiving this because you were mentioned.Message ID: @.*> @0xSage <https://github.com/0xSage> - *Do new windows/views pop up (if so, then no go)?* => No, all the commands above run the installer in headless mode. I tested them, and they work as expected. - *Is there an increased update wait time?* => We use a network installation but disable the post-install script that pulls cortex.llamacpp from the internet. Instead, it only pulls the new binaries, which are less than 30MB. I tested this, and the update process is smooth and quick. — Reply to this email directly, view it on GitHub <#1415 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQVWFCCKSGV563F4TFYVOKDZ3RYRDAVCNFSM6AAAAABPJ5KC3WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJSGY2TEOJSHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[hiento09]

Hi @dan-homebrew @0xSage , we need to finalize the binary file name of the cortex server by tomorrow so that @vansangpfiev and I can proceed with splitting the binary and updating the installer. Could you help us finalize it? Thanks!

My proposal:

• cortex-server-nightly
• cortex-server-beta
• cortex-server

[vansangpfiev]

We also need to handle the case that user using v1.0.0 update cortex via cortex update command.
I would like to propose an approach:

• User run cortex update on v1.0.0 → only cortex binary (which is CLI in v1.0.1) is updated
• When user run next command, cortex CLI checks if cortex-server exists, if not, output message to ask user to download cortex-server. Something like: cortex v1.0.1 requires cortex-server, to install, run: cortex update --server

cc: @dan-homebrew @gabrielle-ong @hiento09

[gabrielle-ong]

Success Criteria (windows, max, linux)

• Installer: Installer installs 2 different binaries
• Uninstaller: Uninstalls 2 different binaries
• Cortex update replaces the app, installer, uninstaller and binary file (without installing cortex.llamacpp)
• all CLI commands will start API server, if not running
• Edge case: User run cortex update on v1.0.0 → prompted to install cortex-server as with 1.0.1

[gabrielle-ong]

@hiento09 and @vansangpfiev
Can I check if anything missing from this success criteria for the Separate CLI & API epic?

Success Criteria (windows, max, linux)

Installer: Installer installs 2 different binaries
Uninstaller: Uninstalls 2 different binaries
Cortex update replaces the app, installer, uninstaller and binary file (without installing cortex.llamacpp)
all CLI commands will start API server, if not running
Edge case: User run cortex update on v1.0.0 → prompted to install cortex-server as with 1.0.1

[vansangpfiev]

@gabrielle-ong

all CLI commands will start API server, if not running

cortex pull and cortex engines install do not need to start API server for now
Others Success Criteria are correct. @hiento09 Please help to double check.

[hiento09]

@gabrielle-ong The installer and uninstaller meet the success criteria. Sang and I worked on the same PR here #1499 . You can test it from the nightly version 183 where our PR was merged.

[gabrielle-ong]

Thanks @hiento09 and @vansangpfiev, marking as complete
Tracking the followup task #1523 (cortex pull and cortex engines to use API server)

Successfully installed mac, windows, linux
Sucessfully uninstalled cortex-nightly and cortex-server-nightlyEdge case successfully encountered last week, rancortex update --server`