Decouple Timer triggering from general stream processing

[ChrisKujawa]

Description

Found at the Chaos day https://github.com/zeebe-io/zeebe-chaos/edit/master/chaos-days/2020-07-09/summary.md

We currently have the issue that we trigger timers in the same actor as the stream processing runs. This means we interrupt the normal processing to write new commands. This should be decoupled to streamline the processing. I can imagine that we can trigger the timers and also other things like timeout jobs, in a separate actor, which then writes to the dispatcher.

This would mean the stream processor can just continue with consuming new records. I think this will make the throughput more stable in case we have more timers. This cam also be done for jobs or messages etc.

[npepinpe]

Duplicate of #4783

[ChrisKujawa]

Wouldn't say this is a duplicate

[npepinpe]

Fair enough, let's talk about it in the morning

[npepinpe]

Alright, so from my pov they're duplicate as they describe the same issue but propose a different approach: triggering timers can potentially slow down stream processing. Maybe you can elaborate what the other issue isn't covering, or how it's different?