Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

impl RecordBatchStream for QueryResults #37

Conversation

alamb
Copy link

@alamb alamb commented Apr 22, 2022

While trying out apache#2226 in IOx, I found it would be super convenient to have QueryResults be a RecordBatchStream (so it has the same interface as the current executions)


/// Keep a reference to the [`QueryTask`] so it isn't dropped early
#[allow(unused)]
query: Arc<QueryTask>,
}

impl Stream for QueryResults {
type Item = Result<RecordBatch>;
type Item = ArrowResult<RecordBatch>;
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is one quirk I was hoping to take the opportunity to fix, that DataFusion streams don't return a DataFusion error... It isn't the end of the world, but I'm curious how important this aspect is?

Copy link
Author

@alamb alamb Apr 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is unimportant in the overall design sense

I think it is critically important from the "minimize changes needed to existing code" sense

@tustvold
Copy link
Owner

tustvold commented May 3, 2022

Implemented as part of apache@505e880

@tustvold tustvold closed this May 3, 2022
@alamb alamb deleted the alamb/scheduler_record_batch branch August 8, 2023 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants