Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move the subgraph task to the blocking thread pool #956

Merged
merged 1 commit into from
May 22, 2019

Conversation

leoyvens
Copy link
Collaborator

This is probably the busiest task in indexing nodes and it blocks on the DB on many of its steps. Locally I could not reliably measure whether it reduced contention, but still seems worth seeing what happens on the hosted service. Even if we can't see a change, in principle this is a correct thing to do.

Part of #937

It has a lot of database interactions, this should reduce
the amount of thread pool contention in indexing nodes.
@leoyvens leoyvens requested a review from a team May 21, 2019 15:17
@leoyvens leoyvens self-assigned this May 21, 2019
@lutter
Copy link
Collaborator

lutter commented May 21, 2019

Looks good in general, and I agree it's worth trying out. Just to help me understand: does this make it so that everything to do with subgraph indexing runs on the blocking pool? Are there any downsides to that?

And: how will we know that this change is an improvement once it gets put into production?

@leoyvens
Copy link
Collaborator Author

@lutter It makes it so that task, which is the driver for subgraph indexing, runs in the blocking pool. The downside is that the interaction with the reactor (to park/unpark) requires cross-thread communication, which is neglibile for us. Once tokio-rs/tokio#588 is done, there will be a more convenient way of putting tasks in a blocking pool, and then we can experiment with putting all tasks which may block in the blocking pool.

We can compare the frequency of "Possible contention in tokio threadpool" warnings to tell if it helped, which isn't very scientific but it's what we have.

@leoyvens leoyvens merged commit aa24e74 into master May 22, 2019
@leoyvens leoyvens deleted the leo/move-subgraph-task-to-blocking-pool branch May 22, 2019 15:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants