CPU increase on main process after 1.57.1 -> 1.58.1 upgrade (fetch_events background task)

[Fizzadar]

Two days ago we rolled out v1.58.1 to production and ever since there's been a substantial CPU increase on the main process doing fetch_events background task which I spotted today:

There's also an increased memory usage overall which is what led us to discover this. Unsure what this is doing exactly, any pointers?

Wonder if this is related to #12778 at all?

[bradtgmurray]

Something in Synapse now wants a lot more events, but who?

[bradtgmurray]

3201863 and 17d99f7 are two new callers of get_events_as_list in 1.58, but neither one makes sense I think (one we can't see the sync_partial_state_room background job in grafana, the other I think should be running on our worker that's handling /messages requests I think?)

[babolivier]

What caches do the purple and red graphs correspond to?

[bradtgmurray]

What caches do the purple and red graphs correspond to?

Apologies, they're all just *getEvent* across multiple restarts of the synapse-0 (main worker) process

[babolivier]

I see, thanks. Could you look at the "Cache eviction rate" graph for the *getEvent* (size) label and see if it reports many size-related evictions?

[bradtgmurray]

[Fizzadar]

OK I believe I have got to the bottom of this - leftover from the NULL character fix which meant the update user directory background process was constantly failing and restarting, never updating the stream position.

Reverting the change in https://gitlab.com/beeper/synapse/-/commit/b5874621bea941f1aee59d728a13cbb17a37c438 has now stopped the constant memory growth.

The fetch event calls are still ongoing but I believe this is just the main process running through the huge gap in stream position going the background update, and should clear in the next hour or two. Will confirm and close this out.

[Fizzadar]

Confirmed this is now fixed by the above revert!

It would be good to investigate ways to better attribute nested background task CPU (if this had shown as user directory background task CPU it'd be super quick to identify), but will leave that for another issue.