[BCFR-899] MaxLogsKept implementation

[reductionista]

BCFR-899

Motivation

Presently, LogPoller supports only time-based retention, via the Retention field in the filters passed to RegisterFilter. The MaxLogsKept field was added earlier in anticipation of the need for also supporting recency-count based retention. One example of a case where time based retention is risky is the Transmit event in the OCR Contract Transmitter. No matter how long the retention period is set to, there's a chance the node will be down for longer than that, and miss logs when it comes back up. This would make a bad situation even worse, because the transmit event would never be picked up at all.

Solution

This implements the MaxLogsKept feature in LogPoller. When specified, this field tells LogPoller it's okay to prune logs matching a filter if there are at least MaxLogsKept more recent matching logs in the db.
In the example above, this avoid storing any more logs than needed while always having the latest transmit event available. In this case, older transmit events are no longer relevant if there is a more recent one.
In general, this should be just as useful for anything accessed only via ChainReader's GetLatestValue() method rather than QueryKey().

A log may be pruned either because it's too old in terms of time or in terms of the number of logs being saved. It need not satisfy both theRetention and MaxLogsKeptcriteria in order to get pruned.

Testing

This was tested by changing the MaxLogsKept setting on the ContractTransmitter filter passed to LogPoller from 0 to 1, and running the CCIP load tests.

Without paging, this query is one of the most cpu intensive. Similar to the DeleteExpiredLogs query, it must go though every row of the logs table unless LogPrunePageSize is set to non-zero. But it's slower than DeleteExpiredLogs because on top of that, it also has to group, sort, and count every log in each group in order to figure out how many there are and which ones are the excess logs eligible for deletion. Also similar to DeleteExpiredLogs, the final step is to merge together all the results for logs matching multiple filters by making sure that no log is dropped unless ALL of its matching filters agree that it came before their own MaxLogsKept threshold was hit.

Without paging, the SelectExcessLogs query grew in median duration linearly as the number of logs in the table increased. After a few hours it was taking longer than the Insert queries (and any other queries) and started causing timeouts to occur, not just for itself but for other queries as well. It eventually got to a point where the p90 & p99 charts were continuously in the 4-5s range, generating many critical errors as well as a lot of backlogged queries waiting on connections.

With a paging size of 4000, you could see that the query durations grew linearly at first with the size of the table, and then leveled off at slightly more than the insert queries. It still resulted in a couple fairly large bursts of critical errors (query timeouts) during the heaviest cpu usage.

A paging size of 1000-2000 worked much better. There were a much smaller # of timeouts, and only during a brief window of time. The p99 durations were noticeably lower than the insert query durations, aside from some very brief but high spikes which we believe are due to an unrelated bug (which has been fixed in Chainlink repo, but hasn't been back posted to CCIP yet) in not letting go of connections quickly enough (due to sql logging).
Aside from these spikes, the charts look pretty healthy... so we should retest once that bug is backported but before the MaxLogsKept feature is enabled.

Another scenario tested was the performance of the query when all filters have MaxLogsKept=0. This was tested both with and without the code which skips the query unless there is at least one filter with MaxLogsKept > 0. Obviously, skipping the query took it off the chart entirely... so the rest of the chart looked the same as before this PR. Without slipping the query, it was slightly faster than the MaxLogs=1 case, but not a very significant reduction. Based on this it was decided it did make sense to disable the pruning until this feature starts being used by at least 1 filter.

With MaxLogsKept=1:

With MaxLogsKept=0 (disabled):

For both of these tests, the pruning of unmatched logs--which is also an expensive query--has been increased in frequency from every 20 ticks to every 4 ticks, to see what it looks like under the most severe conditions while they're both happening at once (with every 20 ticks, the test usually completes before the first one happens). On the MaxLogsKept=1 charts, SelectUnmatchedLogs query durations show up as cyan and SelectExcessLogs query durations as bright yellow. On the MaxLogsKept=0 charts, SelectUnmatchedLogs is pale yellow (and SelectExcessLogs does not run--as desired). LogPrunePageSize was set to 1000 for both.

[dhaidashenko]

Impressive query 🤯!
Left a couple of nits

[cl-sonarqube-production]

Quality Gate passed

Issues
0 New issues
1 Fixed issue
0 Accepted issues

Measures
0 Security Hotspots
70.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube