Bring gzip compression for API server to GA

[smarterclayton]

This KEP proposes how we can move the API server alpha api compression feature to GA by correcting the previous implementation and preventing performance regression.

[smarterclayton]

@liggitt @kubernetes/sig-api-machinery-api-reviews @kubernetes/sig-scalability-api-reviews

Implementation in kubernetes/kubernetes#77449 addresses previous issues, this summarizes the choices. The implementation mitigates all previously known issues, but has a potential impact on scale at the very largest clusters which we can tune by choosing a higher limit if necessary.

[fejta-bot]

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

[liggitt]

rationale and approach seem reasonable. just one (mostly semantic) question around GA vs beta->GA, then lgtm

[smarterclayton]

Updated

[lavalamp]

Haven't read this yet, a quick thought before I forget--

How does this work with watches? It'd be pretty great if we did the gzip compression once and then dropped those bytes on the wire directly for every watcher of that resource, like the next level version of @wojtek-t's #924. I'm not sure if that is feasible off the top of my head but it'd be awesome if we could make it work.

[wojtek-t]

How does this work with watches? It'd be pretty great if we did the gzip compression once and then dropped those bytes on the wire directly for every watcher of that resource, like the next level version of @wojtek-t's #924. I'm not sure if that is feasible off the top of my head but it'd be awesome if we could make it work.

At least initially we decided to only use compression for large-enough list requests. Compressing everything was too expensive with not that huge gain. And I wouldn't like to block this feature on ensuring that this will work for sure.

[smarterclayton]

Based on recent tests gzip on small responses is a significant CPU and p99 regression. So I think watches are probably not going to be good candidates for compression in any near future, especially given that without more clever transfer-encoding: gzip behavior (which requires client side changes) to preserve a gzip dictionary across individual events we aren't going to see much benefit.

I can add that to this - "Experimentation with small gzip behavior on GET requests has shown significant p99 latency and CPU use increases, and given that watch gzip requires a more complex client implementation in order to realize delta benefits, we have no near term plans to extend this to watches. Nothing in the current approach limits our ability to explore that function in the future."

[liggitt]

/lgtm

[smarterclayton]

@lavalamp can you approve / give last comments?

[smarterclayton]

Ping

[lavalamp]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lavalamp, liggitt, smarterclayton

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• keps/sig-api-machinery/OWNERS [lavalamp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment