-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Track memory usage in Non Limited Operators #1569
Comments
I want to propose adding a new kind of Metric: #[derive(Debug, Clone)]
pub struct Gauge {
/// value of the metric gauge
value: std::sync::Arc<AtomicUsize>,
}
impl Gauge {
/// create a new gauge
pub fn new() -> Self {
Self {
value: Arc::new(AtomicUsize::new(0)),
}
}
/// Add `n` to the metric's value
pub fn add(&self, n: usize) {
// relaxed ordering for operations on `value` poses no issues
// we're purely using atomic ops with no associated memory ops
self.value.fetch_add(n, Ordering::Relaxed);
}
/// Substract `n` from the metric's value
pub fn sub(&self, n: usize) {
}
/// set the metric's value to `n`
pub fn set(&self, n: usize) {
}
/// Get the current value
pub fn value(&self) -> usize {
self.value.load(Ordering::Relaxed)
}
} And by adding Besides, a |
Yeah makes sense to me, i am actually surprised we don't have gauge type already. Do we really need the |
|
I agree getting a Gauge sounds like a good idea. thanks for the writeup @yjshen |
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Ensure that physical
ExecutionPlan
nodes properly report their memory usage so DataFusion's can properly stay under its memory budgetDescribe the solution you'd like
Tracking
memory consumers (api in Initial MemoryManager and DiskManager APIs for query execution + External Sort implementation #1526)Describe alternatives you've considered
N/A
Context
This is follow on work from the great PR from @yjshen in #1526 and part of the story of limiting memory used by DataFusion #587
The text was updated successfully, but these errors were encountered: