Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Implement TensorRT-LLM Extension #2323

Closed
imtuyethan opened this issue Mar 12, 2024 · 6 comments
Closed

feat: Implement TensorRT-LLM Extension #2323

imtuyethan opened this issue Mar 12, 2024 · 6 comments
Assignees
Labels
P0: critical Mission critical type: feature request A new feature
Milestone

Comments

@Van-QA
Copy link
Contributor

Van-QA commented Mar 13, 2024

@louis-jan, please help us resolve these issues:

  1. The extension still appears in Jan Windows on a PC with no GPU ❌ clicking on install doesn't install anything + no toast message / warning to users, nothing happen.
    image

  2. Install UI on dark mode:
    image

  3. Missing label on the new model:
    image

  4. Unclear error toast title ❌ only "Failed"
    image

@namchuai namchuai moved this from Planned to In Progress in Menlo Mar 14, 2024
@Van-QA
Copy link
Contributor

Van-QA commented Mar 14, 2024

  1. Missing recommendation label ❌
    image

  2. Need consistency in the TensorRT label ❌
    image

@namchuai
Copy link
Contributor

issue 5&6 is addressed in the PR #2346

@Van-QA
Copy link
Contributor

Van-QA commented Mar 14, 2024

  1. TensorRT model does not functioning with RAG or with Stream off ❌
    image

@namchuai namchuai moved this from In Progress to In Review in Menlo Mar 14, 2024
@namchuai namchuai moved this from In Review to QA in Menlo Mar 14, 2024
@Van-QA
Copy link
Contributor

Van-QA commented Mar 14, 2024

  1. Swagger chat completion is incompatible with TensorRT model for the local API server ❌
image
  1. On Windows with no GPU, the incompatible value is incorrect ❌
    image

  2. Extra space in the extension name:
    image

@Van-QA
Copy link
Contributor

Van-QA commented Mar 19, 2024

Tested and looking good using Jan v0.4.8-333 ✅

@Van-QA Van-QA closed this as completed Mar 19, 2024
@github-project-automation github-project-automation bot moved this from QA to Done in Menlo Mar 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P0: critical Mission critical type: feature request A new feature
Projects
Archived in project
Development

No branches or pull requests

4 participants