Reduce thread contention in subgraph indexing

[Jannis]

No description provided.

[leoyvens]

Plan as outlined in #856 (comment):

• Set the tokio threadpool size to 100.
• Solve Move slow db interactions to tokio blocking pool #905 by moving all store operations into a separate, blocking threadpool, and have them return futures. This is not hard but it's laborious and will have a big diff.
• Try going back to the default threadpool size.

[leoyvens]

Since increasing the size I still see 1ms and 10ms contention warnings, but no 100ms which is an improvement, contention seems to have shifted to the DB connection pool which was expected. Still the situation in tokio is not great, probably due to the issue described in tokio-rs/tokio#426 (comment).

[Jannis]

What's the latest here? Didn't we add the blocking threadpool recently?

[leoyvens]

I believe we've sufficiently mitigated this issue that it shouldn't have any practical effect, but the solution is not ideal, so I'd like to keep this open. My explanation for the thread pool contention warnings that can still be observed sometimes is that half of them are due do this crossbeam bug (fix isn't released yet) and the other half is simply that there is contention for the CPUs, I haven't confirmed this yet, but it would be an ops concern, so I'm removing the label from this issue.