[Event Service] Intermittent high CPU usage

[boreq]

Sometimes the server experiences high CPU usage which doesn't seem to be reflected in the reported high CPU usage of individual processes running on it. For example the 5/10/15 minute load will be extremely high while the individual processes all stay at <10% of CPU usage.

Could be something I/O related or maybe related to goroutines or mutexes around the database connection. After a while (sometimes hours) the service is able to continue normally which points to the DB connection.

[boreq]

Fixes which may in some way contribute towards fixing this so far:

• Avoid loading events for no reason #44
• Verify events as late as possible #45
• Reduce memory usage and number of goroutines when sending events #54
• Optimize downloader #55
• Allow transactions to timeout #56

Looking at the graphs the service no longer hangs in this way.

[boreq]

I think it still sometimes happens e.g. on 2023-12-04 09:10.

[boreq]

Helper task:

• Add pyroscope metrics #59

[boreq]

New pull request:

• Time window based replication #60