Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NMINST tutorial not utilizing full GPU #57

Open
3 of 12 tasks
timcheck opened this issue Mar 22, 2022 · 2 comments
Open
3 of 12 tasks

NMINST tutorial not utilizing full GPU #57

timcheck opened this issue Mar 22, 2022 · 2 comments
Assignees
@timcheck @bamsumit
Labels
1-feature New feature or request performance

Comments

@timcheck
Copy link
Contributor

Objective of issue:

  • Increase GPU utilization.

Lava DL version:

  • 0.3.0 (feature release)
  • 0.2.1 (bug fixes)
  • 0.2.0 (current version)
  • 0.1.2

Lava version:

  • 0.4.0 (feature release)
  • 0.3.1 (bug fixes)
  • 0.3.0 (current version)
  • 0.2.0
  • 0.1.2

I'm submitting a ...

  • bug report
  • feature request
  • documentation request

Current behavior:

  • GPU utilization is not high enough.

Expected behavior:

  • Would like GPU utilization to be >50%.

Steps to reproduce:

  • Run the notebook and monitor GPU utilization with nvidia-smi.
@timcheck timcheck added 1-bug Something isn't working 1-feature New feature or request labels Mar 22, 2022
@knurzl
Copy link

knurzl commented Dec 15, 2022

I have the same issue when working with the assistant but not with my own training loop without the assistant. Then the GPU utilization is nearly 100%, with the assistant only about 1/3. I am using lava-dl 0.3.2 and lava-nc 0.5.1.

@bamsumit
Copy link
Contributor

@knurzl please look at this PR for the fix.

@mathisrichter mathisrichter added 0-needs-review performance and removed 1-feature New feature or request labels Jan 9, 2023
@bamsumit bamsumit added 1-feature New feature or request and removed 1-bug Something isn't working 0-needs-review labels Jan 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@timcheck timcheck

@bamsumit bamsumit

Labels
1-feature New feature or request performance
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mathisrichter @knurzl @timcheck @bamsumit