pkg/kates: fix bug where Accumulator was not coalescing changes mid-watch

[haq204]

Description

The Accumulator struct attempts to coalesce changes into a single snapshot update as a way to do graceful load shedding.
However, while this was the behavior on bootstrap, it didn't always happen mid-watch - each event that was received turned into a single snapshot update, thus not really satisfying this requirement.

We add a new option to batch changes for a specified window interval before sending a snapshot update.
The batching behavior is as follows:

• The Accumulator will receive raw changes up until the window period where it will then send a change, even if new updates are still coming in.
  This is to prevent the potential of a scenario where a change is never sent due to an extremely volatile cluster.
  While there may be a way to be more dynamic in how long to wait before sending this change, this approach is simpler and more predicable.

• If an isolated updated comes in (e.g. last change was submitted an hour ago but the window period is set to 10s), it may not neccessarily wait until the window period before sending change, it can send immediately.

• The default interval is set to 1s to be inline with current change velocity.

• A snapshot update won't be sent until all resources are fully bootstrapped, regardless of what interval is set.
  This is the ensure that the other requirements for the Accumulator are still satisfied.

AMBASSADOR_RECONFIG_MAX_DELAY controls the interval to wait before sending snapshot updates when listening for K8s resources, especially when many resources are updated in quick succession.

Related Issues

Issue is not in this repo.

Testing

Add new test cases.

For performance testing, ran a test that concurrently applies namespaces with each namespace applying 35 deployments, host, and mappings each. Each deployment has 2 replicas. We track the number of snapshot versions pushed. Prevously the number of snapshot versions created when applying 1 namespace was ~118. After setting a 10s interval, the snapshot version reduced to 16. For max concurrent namespaces it would previously start OOMing at 6 concurrent namespaces; with a 10 sec interval it's at least >15. Peak memory usage was 600MB so with a higher memory limit, it can likely support much higher.

Checklist

• I made sure to update CHANGELOG.md.

  Remember, the CHANGELOG needs to mention:

  • Any new features
  • Any changes to our included version of Envoy
  • Any non-backward-compatible changes
  • Any deprecations

• This is unlikely to impact how Ambassador performs at scale - load testing shows xxxx.

  Remember, things that might have an impact at scale include:

  • Any significant changes in memory use that might require adjusting the memory limits
  • Any significant changes in CPU use that might require adjusting the CPU limits
  • Anything that might change how many replicas users should use
  • Changes that impact data-plane latency/scalability

• My change is adequately tested.

  Remember when considering testing:

  • Your change needs to be specifically covered by tests.
    • Tests need to cover all the states where your change is relevant: for example, if you add a behavior that can be enabled or disabled, you'll need tests that cover the enabled case and tests that cover the disabled case. It's not sufficient just to test with the behavior enabled.
  • You also need to make sure that the entire area being changed has adequate test coverage.
    • If existing tests don't actually cover the entire area being changed, add tests.
    • This applies even for aspects of the area that you're not changing – check the test coverage, and improve it if needed!
  • We should lean on the bulk of code being covered by unit tests, but...
  • ... an end-to-end test should cover the integration points

• I updated DEVELOPING.md with any any special dev tricks I had to use to work on this code efficiently - N/A.

• The changes in this PR have been reviewed for security concerns and adherence to security best practices.

[LanceEa]

Quite a few comments, questions and suggestions. I will ping you in the morning to go over a few items to make sure I'm understanding them all correctly.

BTW...awesome to see the improved testing results :)

[LanceEa]

@ddymko - I think it is important for you to have yours eyes on this just so you familiarize yourself with it a little bit.

[haq204]

Going to make some changes after discussions with @LanceEa and @ddymko on expected batching behavior

[LanceEa]

overall looks good based on our discussion. Just a quick question and I agree with David's feedback.

[LanceEa]

lgtm

[ddymko]

@haq204 I think you have to rerun make generate because we cut the 3.2.0-rc.1 yesterday

Otherwise it looks good!

[ddymko]

looks good