Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Verify that tokio 0.2 fixes concurrency bottleneck #1696

Closed
lukesteensen opened this issue Feb 4, 2020 · 4 comments · Fixed by #1922 or #2145
Closed

Verify that tokio 0.2 fixes concurrency bottleneck #1696

lukesteensen opened this issue Feb 4, 2020 · 4 comments · Fixed by #1922 or #2145
Assignees
Labels
domain: performance Anything related to Vector's performance type: enhancement A value-adding code change that enhances its existing functionality. type: tech debt A code change that does not add user value.

Comments

@lukesteensen
Copy link
Member

Our assumption has been that tokio 0.2, with a completely rebuilt scheduler, should no longer suffer from the issue we were seeing at high concurrency levels (see #391). We should verify this assumption and remove our hardcoded concurrency limit if that is true.

This should follow #1695, and we can use pretty much the same plan of attack. The first step should be to remove the current thread limit on a branch. We can then try to reproduce the original issue of plummeting throughput at higher thread counts on the current runtime and see whether or not it still occurs on the 0.2 runtime.

@lukesteensen lukesteensen added type: tech debt A code change that does not add user value. type: performance labels Feb 4, 2020
@binarylogic binarylogic modified the milestones: Improved high concurrency performance, Tech-Debt Payment #1: Move to Tokio 0.3 Feb 22, 2020
@MOZGIII MOZGIII mentioned this issue Feb 25, 2020
7 tasks
@Hoverbear
Copy link
Contributor

Seems related to #1922 ?

@Hoverbear Hoverbear linked a pull request Mar 10, 2020 that will close this issue
7 tasks
@MOZGIII
Copy link
Contributor

MOZGIII commented Mar 10, 2020

It's related. It depends on whether we'll do this assertion at part of #1922, or if we'll do it separately. Technically, it's not a blocker for #1922, and can be addressed separately. If we do validate this as part of #1922, I'll close this.

@MOZGIII
Copy link
Contributor

MOZGIII commented Mar 18, 2020

This was not validated, so reopening it.
Any suggestions on how to approach it?

@MOZGIII MOZGIII reopened this Mar 18, 2020
@binarylogic
Copy link
Contributor

Yep, a test on the test harness with a high cpu instance should do it.

@binarylogic binarylogic modified the milestones: Tech-debt payment #1: Move to Tokio 0.2/Futures 0.3, Improved high concurrency performance Apr 2, 2020
@binarylogic binarylogic added type: enhancement A value-adding code change that enhances its existing functionality. domain: performance Anything related to Vector's performance and removed type: performance labels Aug 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: performance Anything related to Vector's performance type: enhancement A value-adding code change that enhances its existing functionality. type: tech debt A code change that does not add user value.
Projects
None yet
4 participants