diff --git a/keps/sig-api-machinery/4006-transition-spdy-to-websockets/README.md b/keps/sig-api-machinery/4006-transition-spdy-to-websockets/README.md new file mode 100644 index 000000000000..ceb750baffde --- /dev/null +++ b/keps/sig-api-machinery/4006-transition-spdy-to-websockets/README.md @@ -0,0 +1,1228 @@ + +# KEP-4006: Transition from SPDY to WebSockets + + + + + + +- [Release Signoff Checklist](#release-signoff-checklist) +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [User Stories (Optional)](#user-stories-optional) + - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [Background: Streaming Protocol Basics](#background-streaming-protocol-basics) + - [Background: RemoteCommand Subprotocol](#background--subprotocol) + - [Background: API Server and Kubelet UpgradeAwareProxy](#background-api-server-and-kubelet-) + - [Proposal: kubectl WebSocket Executor and Fallback Executor](#proposal--websocket-executor-and-fallback-executor) + - [Proposal: v5.channel.k8s.io subprotocol version](#proposal--subprotocol-version) + - [Proposal: API Server StreamTranslatorProxy](#proposal-api-server-) + - [Future: Kubelet StreamTranslatorProxy](#future-kubelet-) + - [Test Plan](#test-plan) + - [Prerequisite testing updates](#prerequisite-testing-updates) + - [Unit tests](#unit-tests) + - [Integration tests](#integration-tests) + - [e2e tests](#e2e-tests) + - [Graduation Criteria](#graduation-criteria) + - [Alpha](#alpha) + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Version Skew Strategy](#version-skew-strategy) +- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) + - [Feature Enablement and Rollback](#feature-enablement-and-rollback) + - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning) + - [Monitoring Requirements](#monitoring-requirements) + - [Dependencies](#dependencies) + - [Scalability](#scalability) + - [Troubleshooting](#troubleshooting) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) +- [Infrastructure Needed (Optional)](#infrastructure-needed-optional) + + +## Release Signoff Checklist + + + +Items marked with (R) are required *prior to targeting to a milestone / release*. + +- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR) +- [ ] (R) KEP approvers have approved the KEP status as `implementable` +- [ ] (R) Design details are appropriately documented +- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors) + - [ ] e2e Tests for all Beta API Operations (endpoints) + - [ ] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) + - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free +- [ ] (R) Graduation criteria is in place + - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) +- [ ] (R) Production readiness review completed +- [ ] (R) Production readiness review approved +- [ ] "Implementation History" section is up-to-date for milestone +- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] +- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes + + + +[kubernetes.io]: https://kubernetes.io/ +[kubernetes/enhancements]: https://git.k8s.io/enhancements +[kubernetes/kubernetes]: https://git.k8s.io/kubernetes +[kubernetes/website]: https://git.k8s.io/website + +## Summary + + + +Some Kubernetes clients need to communicate with the API Server using a bi-directional +streaming protocol, instead of the standard HTTP request/response mechanism. A streaming +protocol provides the ability read and write arbitrary data messages between the +client and server, instead of providing a single response to a client request. +For example, the commands `kubectl exec`, `kubectl attach`, `kubectl port-forward`, +and `kubectl cp` all benefit from a bi-directional streaming protocol. Currently, +the bi-directional streaming solution for these `kubectl` commands is SPDY/3.1. For +the communication leg between `kubectl` and the API Server, this enhancement transitions +the bi-directional streaming protocol to WebSockets from SPDY/3.1 for three `kubectl` +commands: `exec`, `attach`, and `cp`. + +## Motivation + + + +The SPDY streaming protocol has been deprecated for around eight years, and by now +many proxies, gateways, and load-balancers do not support SPDY. Our effort to modernize +the streaming protocol between Kubernetes clients and the API Server using WebSockets +is necessary to enable the aforementioned intermediaries. WebSockets is a currently +supported standardized protocol (https://www.rfc-editor.org/rfc/rfc6455) that guarantees +compatibility and interoperability with the different components and programming +languages. Finally, WebSockets is preferrable to HTTP/2.0 because the updated HTTP +standard does not support streaming well. The decision to forego HTTP/2.0 is discussed +at greater length in the [Alternatives Section](##Alternatives). + +### Goals + + + +1. Transition the bi-directional streaming protocol from SPDY/3.1 to WebSockets for +`kubectl exec`, `kubectl attach`, and `kubectl cp` for the communication leg +between `kubectl` and the API Server. + +### Non-Goals + + + +1. We do not intend to initially transition the current SPDY bi-directional streaming +protocol for `kubectl port-forward`. This command requires a different subprotocol +than the previously stated three `kubectl` commands (which use the `RemoteCommand` +subprotocol). We intend to transition `kubectl port-forward` in a future release. + +2. We do not initially intend to modify *any* of the SPDY communication paths between +other cluster components other than `kubectl` and the API Server. In other words, +the current SPDY communication paths between the API Server and Kubelet, or Kubelet +and the Container Runtime will not change. + +3. We will not make *any* changes to current WebSocket based browser/javascript clients. + +## Proposal + + + +Currently, the bi-directional streaming protocols (either SPDY or WebSockets) are +initiated from clients, proxied by the API Server and Kubelet, and terminated at +the Container Runtime (e.g. containerd or CRI-O). This enhancement proposes to 1) +modify `kubectl` to request a WebSocket based streaming connection, and to 2) modify +the current API Server proxy to translate the `kubectl` WebSockets data stream to +a SPDY upstream connection. In this way, the cluster components upstream from the +API Server will not need to be changed. + +### User Stories (Optional) + + + + + +N/A + +### Notes/Constraints/Caveats (Optional) + + + +N/A + +### Risks and Mitigations + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +## Design Details + + + +**Current SPDY Streaming Architectural Diagram** + +![Current SPDY Streaming Architectural Diagram](./spdy-architechtural-diagram.png) + +### Background: Streaming Protocol Basics + +`kubectl` bi-directional streaming connections are by created by upgrading an +initial HTTP/1.1 request. By adding two headers (`Connection: Upgrade`, `Upgrade: SPDY/3.1`), +the request can initiate the streaming upgrade. And when the response returns status +`101 Switching Protocols` signalling success, the connection can then be kept open +for subsequent streaming. An example of an upgraded HTTP Request/Response for `kubectl exec` +could look like: + +**HTTP Request** +``` +POST /api/v1/…/pods/nginx/exec?command=... HTTP/1.1 +Connection: Upgrade +Upgrade: SPDY/3.1 +X-Stream-Protocol-Version: v4.channel.k8s.io +X-Stream-Protocol-Version: v3.channel.k8s.io +X-Stream-Protocol-Version: v2.channel.k8s.io +X-Stream-Protocol-Version: v1.channel.k8s.io +``` + +**HTTP Response** +``` +HTTP/1.1 101 Switching Protocols +Connection: Upgrade +Upgrade: SPDY/3.1 +X-Stream-Protocol-Version: v4.channel.k8s.io +``` + +If the upgrade is successful, one of the requested subprotocol versions is chosen +and returned in the response. In this instance, the chosen version of the subprotocol +is: `v4.channel.k8s.io`. + +### Background: `RemoteCommand` Subprotocol + +![Remote Command Subprotocol](./remote-command-subprotocol.png) + +Once the connection is upgraded to a bi-directional streaming connection, the +client and server can exchange data messages. These messages are interpreted with +agreed upon standards which are called subprotocols. The three `kubectl` commands +(`exec`, `attach`, and `cp`) communicate using the `RemoteCommand` subprotocol. Basically, +this subprotocol provides command line functionality from the client to a running +container in the cluster. By multiplexing `stdin`, `stdout`, `stderr`, and `tty` +resizing over a streaming connection, this subprotocol supports clients executing +and interacting with commands executed on a container in the cluster. An example of +`kubectl exec` running the `date` command on an `nginx` pod/container is: + +``` +$ kubectl exec nginx -- date +Tue May 16 03:34:04 PM PDT 2023 +``` + +### Background: API Server and Kubelet `UpgradeAwareProxy` + +In order to route the data streamed between the client and the container, both the +API Server and Kubelet must proxy these data messages. Both the API Server and the +Kubelet provide this functionality with the `UpgradeAwareProxy`, which is a reverse +proxy that knows how to deal with the connection upgrade handshake. + +### Proposal: `kubectl` WebSocket Executor and Fallback Executor + +This enhancement proposes adding a `WebSocketExecutor` to `kubectl`, implementing +the WebSocket client using a new subprotocol version (`v5.channel.k8s.io`). Additionally, +we propose creating a `FallbackExecutor` to address client/server version skew. The +`FallbackExecutor` first attempts to upgrade the connection to WebSockets +version five (`v5.channel.k8s.io`) with the `WebSocketExecutor`, then falls back +to the legacy `SPDYExecutor` version four (`v4.channel.k8s.io`), if the upgrade is +unsuccessful. Note that this mechanism can require two request/response trips instead +of one. While the fallback mechanism may require an extra request/response if the +initial upgrade is not successful, we believe this possible extra roundtrip is justified +for the following reasons: + +1. The upgrade handshake is implemented in low-level SPDY and WebSocket libraries, +and it is not easily exposed by these libraries. If it is even possible to modify +the upgrade handshake, the added complexity would not be worth the effort. +2. The streaming is already IO heavy, so another roundtrip will not substantially +affect the perceived performance. +3. As releases increment, the probablity of a WebSocket enabled `kubectl` communicating +with an older non-WebSocket enabled API Server decreases. + +### Proposal: `v5.channel.k8s.io` subprotocol version + +As previously mentioned, we propose incrementing the subprotocol version from four +to five: `v5.channel.k8s.io`. This version will indicate to upstream components +that `kubectl` is interested in using WebSockets between the client and the API Server +if it is supported. + +### Proposal: API Server `StreamTranslatorProxy` + +![Stream Translator Proxy](./stream-translator-proxy-2.png) + +Currently, the API Server role within client/container streaming is to proxy the +data stream using the `UpgradeAwareProxy`. This enhancement proposes to modify the +SPDY data stream between `kubectl` and the API Server by conditionally adding a +`StreamTranslatorProxy` at the API Server. If the request is for a WebSocket upgrade +of version `v5.channel.k8s.io`, the `UpgradeAwareProxy` will delegate to the +`StreamTranslatorProxy`. This translation proxy terminates the WebSocket connection, +and de-multiplexes the various streams in order to pass the data on to a SPDY connection, +which continues upstream (to Kubelet and eventually the container runtime). + +### Future: Kubelet `StreamTranslatorProxy` + +The eventual plan is to incrementally transition all SPDY communication legs to WebSockets. +After the WebSocket communication leg from `kubectl` to the API Server is proven +to work, the next communication leg to transition is the one from the API Server +to the Kubelet. Both the API Server and the Kubelet stream data messages using the +`UpgradeAwareProxy`. Since the initial plan is to modify the `UpgradeAwareProxy` +to include the `StreamTranslatorProxy`, it will be straighforward to transition +this next communication leg by integrating the `StreamTranslatorProxy` the same as +in the API Server. + +The final communication leg to transition from SPDY to WebSockets will be the one +from Kubelet to the Container Runtimes. This effort will be more work, since it will +require modifying not just Kubelet, but **all** Container Runtimes. + +### Test Plan + + + +[X] I/we understand the owners of the involved components may require updates to +existing tests to make this code solid enough prior to committing the changes necessary +to implement this enhancement. + +##### Prerequisite testing updates + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +##### Unit tests + + + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +- ``: `` - `` + +##### Integration tests + + + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +- `: ` + +##### e2e tests + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +- `: ` + +### Graduation Criteria + + + +#### Alpha + +``` +<<[UNRESOLVED @seans3]>> +TBD - currently not finished +<<[/UNRESOLVED]>> +``` + +- `WebSocketExecutor` and `FallbackExecutor` completed and functional behind a `kubectl` feature flag. +- `StreamTranslatorProxy` successfully integrated into the `UpgradeAwareProxy` behind an API Server feature flag. +- Initial unit tests completed and enabled +- Initial integration tests completed and enabled +- Initial e2e tests completed and enabled +- Additional SPDY integration and e2e tests for components other than `kubectl` to +ensure this current functionality is not adversely affected. + +### Upgrade / Downgrade Strategy + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +### Version Skew Strategy + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + + + +## Production Readiness Review Questionnaire + + + +### Feature Enablement and Rollback + + + +###### How can this feature be enabled / disabled in a live cluster? + + + +``` +<<[UNRESOLVED @seans3]>> +TBD - currently not finished +<<[/UNRESOLVED]>> +``` + +- [X] Feature gate (also fill in values in `kep.yaml`) + - Feature gate name: ClientRemoteCommandWebsockets + - Components depending on the feature gate: kubectl, API Server +- [ ] Other + - Describe the mechanism: TODO + - Will enabling / disabling the feature require downtime of the control + plane? Yes. The API Server would have to be restarted to change the value of + the feature flag. `kubectl`, however, would only need an environment variable + to change the value of the feature flag, so it would not affect the control + plane. + - Will enabling / disabling the feature require downtime or reprovisioning + of a node? No. + +###### Does enabling the feature change any default behavior? + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)? + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +###### What happens if we reenable the feature if it was previously rolled back? + +###### Are there any tests for feature enablement/disablement? + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +### Rollout, Upgrade and Rollback Planning + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +###### How can a rollout or rollback fail? Can it impact already running workloads? + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +###### What specific metrics should inform a rollback? + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested? + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.? + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +### Monitoring Requirements + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +###### How can an operator determine if the feature is in use by workloads? + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +###### How can someone using this feature know that it is working for their instance? + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +- [ ] Events + - Event Reason: +- [ ] API .status + - Condition name: + - Other field: +- [ ] Other (treat as last resort) + - Details: + +###### What are the reasonable SLOs (Service Level Objectives) for the enhancement? + + + +###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service? + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +- [ ] Metrics + - Metric name: + - [Optional] Aggregation method: + - Components exposing the metric: +- [ ] Other (treat as last resort) + - Details: + +###### Are there any missing metrics that would be useful to have to improve observability of this feature? + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +### Dependencies + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +###### Does this feature depend on any specific services running in the cluster? + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +### Scalability + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +###### Will enabling / using this feature result in any new API calls? + + + +The proposed design envisions a fallback mechanism when a new `kubectl` communicates +with an older API Server. The client will initially request an upgrade to WebSockets, +but it will fallback to the legacy SPDY if it is not supported. In this version +skew scenario where the client implements the new functionality but the server does +not, there is an extra request/response. Since bi-directional streaming already is +very IO intensive, this extra request/response should not be significant. Additionally, +as releases are incremented, the probability of the version skew will continually +decrease. + + + +###### Will enabling / using this feature result in introducing new API types? + + + +No + +###### Will enabling / using this feature result in any new calls to the cloud provider? + + + +No + +###### Will enabling / using this feature result in increasing size or count of the existing API objects? + + + +``` +<<[UNRESOLVED]>> +TBD - currently not finished +<<[/UNRESOLVED]>> +``` + + + +###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs? + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components? + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +###### Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)? + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +### Troubleshooting + + + +###### How does this feature react if the API server and/or etcd is unavailable? + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +###### What are other known failure modes? + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +###### What steps should be taken if SLOs are not being met to determine the problem? + +## Implementation History + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +## Drawbacks + + + +``` +<<[UNRESOLVED]>> +TBD - currently unimplemented +<<[/UNRESOLVED]>> +``` + +## Alternatives + + + +``` +<<[UNRESOLVED @seans3]>> +TODO - currently not complete +<<[/UNRESOLVED]>> +``` + + +Ironically, HTTP/2.0 is based on SPDY. But the upgraded HTTP standard did not +surface streaming functionality. + +Reasons why HTTP/2.0 is not adequate: + +1. HTTP/2.0 does not support upgrading connections for streaming. +2. HTTP/2.0 does not surface streaming functionality. + +## Infrastructure Needed (Optional) + + + +N/A diff --git a/keps/sig-api-machinery/4006-transition-spdy-to-websockets/kep.yaml b/keps/sig-api-machinery/4006-transition-spdy-to-websockets/kep.yaml new file mode 100644 index 000000000000..aa4543f76850 --- /dev/null +++ b/keps/sig-api-machinery/4006-transition-spdy-to-websockets/kep.yaml @@ -0,0 +1,43 @@ +title: Transition from SPDY to Websockets +kep-number: 4006 +authors: + - "@seans3" +owning-sig: sig-api-machinery +participating-sigs: + - sig-cli +status: provisional +# status other states: |implementable|implemented|deferred|rejected|withdrawn|replaced +creation-date: 2023-05-15 +reviewers: + - "@aojea" + - "@ardaguclu" +approvers: + - "@deads2k" + - "@jpbetz" + +# The target maturity stage in the current dev cycle for this KEP. +stage: alpha + +# The most recent milestone for which work toward delivery of this KEP has been +# done. This can be the current (upcoming) milestone, if it is being actively +# worked on. +latest-milestone: "v1.28" + +# The milestone at which this feature was, or is targeted to be, at each stage. +milestone: + alpha: "v1.28" + beta: "v1.29" + stable: "v1.30" + +# The following PRR answers are required at alpha release +# List the feature gate name and the components for which it must be enabled +feature-gates: + - name: ClientRemoteCommandWebsockets + components: + - kube-apiserver + - kubectl +disable-supported: true + +# The following PRR answers are required at beta release +metrics: + - websocket_fallback_count diff --git a/keps/sig-api-machinery/4006-transition-spdy-to-websockets/remote-command-subprotocol.png b/keps/sig-api-machinery/4006-transition-spdy-to-websockets/remote-command-subprotocol.png new file mode 100644 index 000000000000..cc5264a7e923 Binary files /dev/null and b/keps/sig-api-machinery/4006-transition-spdy-to-websockets/remote-command-subprotocol.png differ diff --git a/keps/sig-api-machinery/4006-transition-spdy-to-websockets/spdy-architechtural-diagram.png b/keps/sig-api-machinery/4006-transition-spdy-to-websockets/spdy-architechtural-diagram.png new file mode 100644 index 000000000000..c6932bb2fa23 Binary files /dev/null and b/keps/sig-api-machinery/4006-transition-spdy-to-websockets/spdy-architechtural-diagram.png differ diff --git a/keps/sig-api-machinery/4006-transition-spdy-to-websockets/stream-translator-proxy-2.png b/keps/sig-api-machinery/4006-transition-spdy-to-websockets/stream-translator-proxy-2.png new file mode 100644 index 000000000000..af4ea5d1f004 Binary files /dev/null and b/keps/sig-api-machinery/4006-transition-spdy-to-websockets/stream-translator-proxy-2.png differ