Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

epic: Basic Automated test for Installation & Inference on compatible Hardware & OS #1147

Closed
1 task
dan-menlo opened this issue Sep 8, 2024 · 4 comments
Closed
1 task
Assignees
Labels
category: tests QA automations, tests P1: important Important feature / fix
Milestone

Comments

@dan-menlo
Copy link
Contributor

dan-menlo commented Sep 8, 2024

Goal

  • We should test for full e2e lifecycle of Cortex's main use case
  • This only covers llama.cpp for v0.1

Existing Work

Existing Bugs

Test Cases

Test Harness

Installation & Uninstallation

Starting

Model Running

  • Successful load of cached model (e.g. tinyllama)
  • Successful inference request of cached model
  • Error messages are tested
  • Successful unloading of model
  • Ensure dylib issue is covered bug: libengine.dylib not found #953

Stopping

  • Successfully stops with no dangling processes

Uninstallation

  • Successfully uninstalls with no dangling files
@dan-menlo dan-menlo added this to Menlo Sep 8, 2024
@dan-menlo dan-menlo converted this from a draft issue Sep 8, 2024
@dan-menlo dan-menlo moved this to In Progress in Menlo Sep 8, 2024
@dan-menlo dan-menlo added the category: tests QA automations, tests label Sep 8, 2024
@dan-menlo dan-menlo changed the title epic: Cortex MVP of automated testing v0.1 epic: Cortex MVP of automated QA Testing v0.1 Sep 8, 2024
@dan-menlo dan-menlo removed their assignment Sep 8, 2024
@dan-menlo dan-menlo changed the title epic: Cortex MVP of automated QA Testing v0.1 epic: Basic Automated test for Installation & Inference on supported Hardware & OS Sep 8, 2024
@dan-menlo dan-menlo changed the title epic: Basic Automated test for Installation & Inference on supported Hardware & OS epic: Basic Automated test for Installation & Inference on compatible Hardware & OS Sep 8, 2024
@freelerobot freelerobot added the P1: important Important feature / fix label Sep 9, 2024
@namchuai
Copy link
Collaborator

For end to end testing. I'm proposing to use pytest.

Advantages

  • Easy testing our CLI and API server
  • Easy to write test as well as generate from AI

Image

@freelerobot freelerobot assigned freelerobot and unassigned namchuai Sep 26, 2024
@freelerobot freelerobot moved this from QA to Planning in Menlo Sep 26, 2024
@freelerobot freelerobot added this to the v0.1.1 milestone Sep 26, 2024
@dan-menlo dan-menlo moved this from Planning to Scheduled in Menlo Sep 29, 2024
@freelerobot
Copy link
Contributor

freelerobot commented Sep 30, 2024

Current Manual QA Checklist

It is a combination of:

Across the following:

  • Windows 11 (online & offline)
  • Ubuntu 24, 22 (online & offline)
  • Mac Silicon OS 14/15 (online & offline)
  • Mac Intel (online & offline)

I'm documenting this for @gabrielle-ong to ensure no tests fall through the cracks and eventually we have coverage across various kinds of tests.

Installation/Uninstallation

  • it should install with network installer
  • it should install with local installer
  • it should uninstall with & without purging the data folder
  • it should gracefully uninstall when server is still running
  • it should reinstall without having conflict issues with existing cortex data folders
  • it should install with correct folder permissions
  • it should install with folders: /engines /models /logs, and .cortexrc, and dbs
  • cortex update should update from ~3-5 versions ago to latest (+3 to 5 bump)
  • cortex update should update from the previous version to latest (+1 bump)
  • cortex update should update from previous stable version to latest (stable checking)
  • it should gracefully update when server is actively running

Basic Commands

  • cortex returns helpful text in a timely* way
  • cortex -v should check output current version and check for updates
  • it should correctly log to cortex-cli

Hardware Detection [WIP]

  • TODO

Server

  • it should start server
  • it should stop server
  • it should correctly log to cortex logs
  • it should return server status ps

Engines

  • it should CRUD engines
  • it should gracefully refine engine installation if interrupted halfway
  • it should update engines
  • it should gracefully handle when users try to CRUD incompatible engines
  • it should run gguf models on llamacpp
  • it should run trtllm models on trt-llm

Model Management

  • it should pull by built in model_ID
  • it should pull by built-in model_id:variant
  • it should pull by HF repo/model ID
  • it should pull by partial HF url
  • it should pull by full HF url (ending in .gguf)
  • it should resume pull after interruption
  • it should CRUD downloaded models
  • it should correctly update state in /models
  • it should import models

Model Running

  • run should download missing models
  • run works on already downloaded models
  • run should autostart server
  • chat works

With Hardware Acceleration

  • it should auto offload max ngl
  • it should correctly detect available GPUs
  • it should gracefully detect missing dependencies/drivers

@dan-menlo dan-menlo removed this from the v0.1.1 milestone Oct 3, 2024
@dan-menlo dan-menlo added this to the v1.0.0 milestone Oct 3, 2024
@gabrielle-ong gabrielle-ong modified the milestones: v1.0.0, v1.0.2 Oct 14, 2024
@gabrielle-ong
Copy link
Contributor

Updated for v1.0.1 (Manual QA & API tests) #1535
to capture changes eg

  • model management syntax
  • recommend default model
  • llama.cpp installed by default
  • API tests
    Closing this issue in favour of iterating the QA list with each update

@github-project-automation github-project-automation bot moved this from Scheduled to Review + QA in Menlo Oct 22, 2024
@gabrielle-ong gabrielle-ong moved this from Review + QA to Completed in Menlo Oct 22, 2024
@dan-menlo
Copy link
Contributor Author

@gabrielle-ong Even though we are choosing to go with a Manual test for now, we should create an open Ticket for an Automated Test and put it in Icebox.

  • We should be careful of "QA Debt", and not let it grow too much

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category: tests QA automations, tests P1: important Important feature / fix
Projects
Archived in project
Development

No branches or pull requests

7 participants