Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

epic: Cortex can detect track free and used VRAM and RAM #1191

Closed
Tracked by #1165
dan-menlo opened this issue Sep 11, 2024 · 1 comment
Closed
Tracked by #1165

epic: Cortex can detect track free and used VRAM and RAM #1191

dan-menlo opened this issue Sep 11, 2024 · 1 comment

Comments

@dan-menlo
Copy link
Contributor

dan-menlo commented Sep 11, 2024 •
edited
Loading

Goal

  • Cortex.cpp can detect total, free and used VRAM and RAM
  • This is pre-requisite to avoid OOM when loading multiple models
  • Pre-requisite for intelligent features (e.g. smart model unloading), e.g. Triton

Related
@dan-menlo dan-menlo added this to Menlo Sep 11, 2024
@dan-menlo dan-menlo converted this from a draft issue Sep 11, 2024
@dan-menlo dan-menlo changed the title epic: Cortex can detect VRAM and RAM use epic: Cortex can detect track free and used VRAM and RAM Sep 11, 2024
@dan-menlo dan-menlo moved this to Planning in Menlo Sep 11, 2024
@dan-menlo dan-menlo removed the status in Menlo Sep 11, 2024
@dan-menlo dan-menlo mentioned this issue Sep 26, 2024
Closed
11 tasks
@dan-menlo
Copy link
Contributor Author

Closing in favor of #1165

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
Menlo
Archived in project
Milestone
No milestone
Development

No branches or pull requests
1 participant
@dan-menlo