Reduce the cpu cycles in runtime.findrunnable

[tanjianfeng]

This code path is indeed very slow in go schedule:

1. runqsteal
2. stopm, which is a simple futex syscall, and exaggerated by sentry (+hundreds of ns).

And this is a generic problem in Go. "the cumulative time spent in runtime.runqsteal was the largest contributor to the cumulative time of runtime.findrunnable".

• runtime: scheduler is slow when goroutines are frequently woken: runtime: scheduler is slow when goroutines are frequently woken golang/go#18237
• runtime: scheduler work stealing slow for high GOMAXPROCS: runtime: scheduler work stealing slow for high GOMAXPROCS golang/go#28808

Historically, people relief this issue by reducing GOMAXPROCS. But in our case, we cannot simplily do that, as smaller GOMAXPROCS leads to larger latency when load becomes larger.

Some unmature ideas:

• Dynamic GOMAXPROCS.
• Avoid the usleep in runqgrab.
• Call runsteal with more strict condition.

Any thoughts?

[prattmic]

We've seen this a lot as well. Reducing this cost is definitely one of our priorities going forward.

[navytux]

Even though it originated in runtime.LockOSThread, this issue looks to be related: golang/go#21827.

[amscanne]

I've created a meta-issue to discuss scalability improvements #2184. There is also a first specific proposal being discussed in #2180.

[prattmic]

@tanjianfeng I am collecting a set of canonical benchmarks to use when working on this problem. If you have any good benchmark workloads that particularly suffer from this that you can share, that would be helpful.

[tanjianfeng]

@tanjianfeng I am collecting a set of canonical benchmarks to use when working on this problem. If you have any good benchmark workloads that particularly suffer from this that you can share, that would be helpful.

Sorry for late response. #2033 is an example in hand.

[github-actions]

A friendly reminder that this issue had no activity for 120 days.

[github-actions]

This issue has been closed due to lack of activity.