Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reuse the IoDispatcher across DataFusion instances #1299

Merged
merged 3 commits into from
Nov 14, 2024

Conversation

gatesn
Copy link
Contributor

@gatesn gatesn commented Nov 14, 2024

No description provided.

@gatesn gatesn requested a review from a10y November 14, 2024 19:34
let rx: flume::Receiver<Box<dyn TokioSpawn + Send>> = rx.clone();

worker_thread
.spawn(move || {
// Create a runtime-per-thread
let rt = tokio::runtime::Builder::new_current_thread()
.thread_name(format!("tokio-dispatch-{tid}"))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tokio overrides the thread name, so have to set it here not above

a10y
a10y previously requested changes Nov 14, 2024
Copy link
Contributor

@a10y a10y left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think the easier fix is to impl Drop to call shutdown, that way the threads get joined

@a10y a10y dismissed their stale review November 14, 2024 19:42

i was wrong

@gatesn gatesn force-pushed the ngates/reuse-dispatcher-datafusion branch from bc13d66 to f5994c1 Compare November 14, 2024 19:47
@gatesn gatesn enabled auto-merge (squash) November 14, 2024 19:47
@gatesn gatesn merged commit 8484445 into develop Nov 14, 2024
12 checks passed
@gatesn gatesn deleted the ngates/reuse-dispatcher-datafusion branch November 14, 2024 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants