Any reason why logical plan is optimized twice?

[aprimadi]

Hello, I am just curious why the logical plan is being optimized twice.

First, it was optimized here:
https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/execution/context.rs#L205

Then it was optimized again when collecting result
https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/execution/dataframe_impl.rs#L147

[houqp]

probably an oversight? the optimize in sql method does look like redundant.

[Dandandan]

I think it's not an oversight: collect is not necessarily called when executing queries, for example when writing the results to disk. As collect loads the result into memory in one thread/partition, this is not always what you want to do.
We might do something to avoid optimizing twice when calling both functions, but maybe it's not a big deal for now?

[houqp]

That's a good point. Perhaps the better structure would be to push optimize into query evaluation/materialization methods like save and collect. This way, we can make sure the engine always execute with an optimized plan regardless whether it's built from sql or plan builder.

[alamb]

his way, we can make sure the engine always execute with an optimized plan regardless whether it's built from sql or plan builder.

I like that plan

[alamb]

I wonder if this ticket is tracking anything useful or if the question has been answered and we can close the ticket?

[houqp]

i think we already got the answer to the question.