Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disk spill metric #11564

Merged
merged 10 commits into from
Oct 8, 2024
Merged

Disk spill metric #11564

merged 10 commits into from
Oct 8, 2024

Conversation

zpuller
Copy link
Collaborator

@zpuller zpuller commented Oct 7, 2024

Adds a task level disk allocation high watermark accumulator to the GpuTaskMetrics.

Also fixes an issue where direct spills from device to disk were not being tracked in the existing Spark disk spill metric.

Signed-off-by: Zach Puller <[email protected]>
Signed-off-by: Zach Puller <[email protected]>
Signed-off-by: Zach Puller <[email protected]>
Signed-off-by: Zach Puller <[email protected]>
Signed-off-by: Zach Puller <[email protected]>
revans2
revans2 previously approved these changes Oct 8, 2024
Copy link
Collaborator

@revans2 revans2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really just some minor nits around logging.

@revans2
Copy link
Collaborator

revans2 commented Oct 8, 2024

build

@zpuller zpuller changed the base branch from branch-24.10 to branch-24.12 October 8, 2024 16:20
@zpuller zpuller dismissed revans2’s stale review October 8, 2024 16:20

The base branch was changed.

Signed-off-by: Zach Puller <[email protected]>
@zpuller
Copy link
Collaborator Author

zpuller commented Oct 8, 2024

build

revans2
revans2 previously approved these changes Oct 8, 2024
Signed-off-by: Zach Puller <[email protected]>
@zpuller
Copy link
Collaborator Author

zpuller commented Oct 8, 2024

build

Signed-off-by: Zach Puller <[email protected]>
@zpuller
Copy link
Collaborator Author

zpuller commented Oct 8, 2024

build

Signed-off-by: Zach Puller <[email protected]>
@zpuller
Copy link
Collaborator Author

zpuller commented Oct 8, 2024

build

@zpuller zpuller merged commit 506d212 into NVIDIA:branch-24.12 Oct 8, 2024
45 checks passed
@zpuller zpuller deleted the disk_spill_metric branch October 9, 2024 16:04
@sameerz sameerz added the task Work required that improves the product but is not user facing label Oct 11, 2024
@zpuller zpuller mentioned this pull request Nov 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
task Work required that improves the product but is not user facing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants