Misc. improvements for priority task processing #2897
Conversation
| 10000, | ||
| 20000, | ||
| 50000, | ||
| 100000, |
There was a problem hiding this comment.
We emit metrics for # of pending tasks in a shard, and suspend task loading if the number is too high. The original 1000 is too low and can't give a good estimation of the # of pending tasks nor can be used to tell if task loading is suspended or not.
| @@ -25,7 +25,7 @@ | |||
| package tasks | |||
|
|
|||
| const ( | |||
| WeightedChannelDefaultSize = 100 | |||
| WeightedChannelDefaultSize = 10000 | |||
There was a problem hiding this comment.
This can be regarded as the host level task buffer. and handle a burst of task submission. The size should be at least bigger than the priority weight. Set it to a larger number so that when there's a burst of tasks, instead of waiting for a periodical reschedule (which happens 3-5s later), buffer the tasks in the scheduler can shorten the e2e latency since they will be processed as soon as possible.
| @@ -316,8 +316,8 @@ func NewConfig(dc *dynamicconfig.Collection, numberOfShards int32, isAdvancedVis | |||
| TimerTaskMaxRetryCount: dc.GetIntProperty(dynamicconfig.TimerTaskMaxRetryCount, 100), | |||
| TimerProcessorEnableSingleCursor: dc.GetBoolProperty(dynamicconfig.TimerProcessorEnableSingleCursor, false), | |||
| TimerProcessorEnablePriorityTaskScheduler: dc.GetBoolProperty(dynamicconfig.TimerProcessorEnablePriorityTaskScheduler, false), | |||
| TimerProcessorSchedulerWorkerCount: dc.GetIntProperty(dynamicconfig.TimerProcessorSchedulerWorkerCount, 200), | |||
| TimerProcessorSchedulerQueueSize: dc.GetIntProperty(dynamicconfig.TimerProcessorSchedulerQueueSize, 10000), | |||
There was a problem hiding this comment.
I originally thought this is the queue size for the weighted channel. But it's the size of the channel that's actually polled by the worker pool. (Task scheduler basically moves tasks from weighted channel to that single channel according to the weight).
If the number is high, then task scheduler simply move tasks to that single channel in the order they are received. And we lost the effect of wrr scheduler.
I will probably remove this config in the future, I don't think there's a need to tune this number.
What changed?
Why?
How did you test it?
Potential risks
N/A, still disabled by default.
Is hotfix candidate?