Limit how often GC happens by time.

[erikjohnston]

Synapse can be quite memory intensive, and unless care is taken to tune
the GC thresholds it can end up thrashing, causing noticable performance
problems for large servers. We fix this by limiting how often we GC a
given generation, regardless of current counts/thresholds.

This does not help with the reverse problem where the thresholds are set
too high, but that should only happen in situations where they've been
manually configured.

Adds a gc_min_seconds_between config option to override the defaults.

Fixes #9890.

[ShadowJonathan]

This can become out-of-bounds if someone specifies [1, 10] or similar accidentally.

Suggested change

	if threshold[i] < counts[i] and MIN_TIME_BETWEEN_GCS[i] < end - _last_gc[i]:
	if threshold[i] < counts[i] and len(MIN_TIME_BETWEEN_GCS) => i and MIN_TIME_BETWEEN_GCS[i] < end - _last_gc[i]:

I'm also wary of mutating a "constant" like this, because it can give mixed signals in the code.

[erikjohnston]

Yeah, mutating is a bit icky, but I can't think of a better way of injecting the values at this point.

This can become out-of-bounds if someone specifies [1, 10] or similar accidentally.

We check for the right format in the config, so I think its fine that this explodes if a developer messes up and sets it incorrectly.

[richvdh]

sorry, I have Opinions about configuration settings.

[richvdh]

I don't know why we're working in seconds here where we use ms literally everwhere else, but lgtm otherwise.

[erikjohnston]

I don't know why we're working in seconds here where we use ms literally everwhere else, but lgtm otherwise.

Where we using it everything is done in seconds (as we don't have access to the standard clocks), so using milliseconds would involve a lot of repeated conversion