Guidance for integrating a progress bar

[joolean]

Hello!

I'm attempting to use Datafusion to implement an embedded query engine for a tool that occasionally needs to perform CPU-intensive, long-running operations, such as sorting a dataset that is too large to fit in memory. The debug logging produced during the execution of the physical plan during these types of operations is useful to me (as a developer) but for my users I would prefer to render a progress bar instead, using something like indicatif.

Before I jump into the Datafusion internals, I'm wondering if anyone has any suggestions as to the best way to go about this. (Do you think it's even possible in the first place?) Are there any existing hooks for progress events? Or extension points for alternate implementations of, say, an ExecutionPlan?

Thanks in advance. And thanks for all the work that's gone into Datafusion! I've found it to be an extraordinarily useful library.

[oersted]

Any progress with this?

[joolean]

I've been poking around the internals of PhysicalPlan implementations, and it seems like it might be possible to do this by wrapping a record batch stream, maybe in combination with some of the metrics that are produced? But I don't have any concrete ideas yet - if you have thoughts, please share!

[joolean]

I think I've got a sketch of a working approach. Here's the general idea:

1. ExecutionPlan offers a couple of relevant functions: statistics which provides metrics for the physical plan; and execute, which produces a stream of record batches that constitute the output of the plan.
2. I've adapted some of the code here: https://github.com/tmpfs/read-progress-stream and created a stream implementation that wraps the record batch stream and invokes a "progress callback"
3. I've created an alternate implementation of ExecutionPlan that wraps a normal plan and delegates every function except for execute, which I'm hijacking to produce a wrapped progress reader stream.
4. I'm using the output of statistics to determine the bounds of the progress bar.

[alamb]

Another possibility is to call metrics on your execution plan https://docs.rs/datafusion/17.0.0/datafusion/physical_plan/trait.ExecutionPlan.html#method.metrics

Using row counts could give you some estimate of how far your query has progressed.

In terms of estimating how much longer a query has to run I think it is tricky in the sense that the input size is often not known exactly, and some of the operators (like HashAgregate and Joins and Sorts) don't produce any output until they have seen all their input