From c2da875982613da9df6a525d10abb07a62a52891 Mon Sep 17 00:00:00 2001 From: Eric Chiang Date: Wed, 13 Dec 2017 17:56:54 -0800 Subject: [PATCH] proposal: external kubectl auth providers --- .../auth/kubectl-exec-plugins.md | 285 ++++++++++++++++++ 1 file changed, 285 insertions(+) create mode 100644 contributors/design-proposals/auth/kubectl-exec-plugins.md diff --git a/contributors/design-proposals/auth/kubectl-exec-plugins.md b/contributors/design-proposals/auth/kubectl-exec-plugins.md new file mode 100644 index 00000000000..0715457c805 --- /dev/null +++ b/contributors/design-proposals/auth/kubectl-exec-plugins.md @@ -0,0 +1,285 @@ +# Out-of-tree client authentication providers + +Author: @ericchiang + +# Objective + +This document describes a credential rotation strategy for client-go using an exec-based +plugin mechanism. + +# Motivation + +Kubernetes clients can provide three kind of credentials: bearer tokens, TLS +client certs, and basic authentication username and password. Kubeconfigs can either +in-line the credential, load credentials from a file, or can use an `AuthProvider` +to actively fetch and rotate credentials. `AuthProviders` are compiled into client-go +and target specific providers (GCP, Keystone, Azure AD) or implement a specification +supported but a subset of vendors (OpenID Connect). + +Long term, it's not practical to maintain custom code in kubectl for every provider. This +is in-line with other efforts around kubernetes/kubernetes to move integration with cloud +provider, or other non-standards-based systems, out of core in favor of extension points. + +Credential rotation tools have to be called on a regular bases in case the current +credentials have expired, making [kubectl plugins](https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/), +kubectl's current extension point, unsuitable for credential rotation. It's easier +to wrap `kubectl` so the tool is invoked on every command. For example, the following +is a [real example]( +https://github.com/heptio/authenticator#4-set-up-kubectl-to-use-heptio-authenticator-for-aws-tokens) +from Heptio's AWS authenticator: + +``` +kubectl --kubeconfig /path/to/kubeconfig --token "$(heptio-authenticator-aws token -i CLUSTER_ID -r ROLE_ARN)" [...] +``` + +Beside resulting in a long command, this potentially encourages distributions to +wrap or fork kubectl, changing the way that users interact with different +Kubernetes clusters. + +# Proposal + +This proposal builds off of earlier requests to [support exec-based plugins]( +https://github.com/kubernetes/kubernetes/issues/35530#issuecomment-256170024), and +proposes that we should add this as a first-class feature of kubectl. Specifically, +client-go should be able to receive credentials by executing a command and reading +that command's stdout. + +In fact, client-go already does this today. The GCP plugin can already be configured +to [call a command]( +https://github.com/kubernetes/client-go/blob/kubernetes-1.8.5/plugin/pkg/client/auth/gcp/gcp.go#L228-L240) +other than `gcloud`. + +## Plugin responsibilities + +Plugins are exec'd through client-go and print credentials to stdout. Errors are +surfaced through stderr and a non-zero exit code. To use the bearer token +`BEARER_TOKEN`, a plugin would print: + +```terminal +$ ./kubectl-example-auth-plugin +BEARER_TOKEN +``` + +Plugin output doesn't use versioned Kubernetes APIs (with kind and apiVersion) +because we expect this plugin to interact with tools that don't natively understand +Kubernetes. For example, the GCP auth provider execs [`gcloud auth`]( +https://github.com/kubernetes/client-go/blob/kubernetes-1.9.0/plugin/pkg/client/auth/gcp/gcp.go#L230) +and parses the output using JSONPath. + +### Caching + +kubectl repeatedly [re-initializes transports](https://github.com/kubernetes/kubernetes/issues/37876) +while client-go transports are long lived over many requests. As a result naive auth +provider implementations that re-request credentials on every request have historically +been slow. + +Initially, plugins will be responsible for caching credentials and knowing when those +credentials expire, likely by keeping a config file similar to existing auth providers. + +Practically today's auth providers depend on 401 HTTP status codes to either detect +misconfigurations (GCP) or credential expiry (KeyStone). In the future, the exec plugin +may have to pass additional information when exec'ing a plugin after a 401. + +It's unclear what mechanism client-go would use to pass this information to the plugin. +See the [_"Open issues"_](#open-issues) section for more discussion. + +## Kubeconfig changes + +The current `AuthProviderConfig` uses `map[string]string` for configuration, which +makes it hard to express things like a list of arguments or list key/value environment +variables. As such, `AuthInfo` should add another field which expresses the `exec` +config. This has the benefit of a more natural structure, but the trade-off of not being +compatible with the existing `kubectl config set-credentials` implementation. + +```go +const ( + AuthInfoExecKindBearerToken = "authentication.alpha.k8s.io/bearer-token" + AuthInfoExecKindClientCert = "authentication.alpha.k8s.io/tls-certificate" +) + +// AuthInfo contains information that describes identity information. This is use to tell the kubernetes cluster who you are. +type AuthInfo struct { + // Existing fields ... + + // Exec is a command to execute which returns credentials to the transport to use. + // +optional + Exec *ExecAuthProviderConfig `json:"exec,omitempty"` + + // ... +} + +type ExecAuthProviderConfig struct { + Command string `json:"command"` + Args []string `json:"args"` + // Env defines additional environment variables to expose to the process. These + // are unioned with the host's environment, with variable values lists here having + // precedence. + Env []ExecEnvVar `json:"env"` + // Kind determines the kind of credentials the command will output to stdout. + // This can be "bearer-token" or "tls-certificate". + Kind string `json:"kind"` + + // TODO: JSONPath options for filtering output. +} + +type ExecEnvVar struct { + Name string `json:"name"` + Value string `json:"value"` + + // TODO: Load env vars from files or from other envs? +} +``` + +This would allow a user block of a kubeconfig to declare the following: + +```yaml +users: +- name: mmosley + user: + exec: + kind: "authentication.alpha.k8s.io/bearer-token" + command: /bin/kubectl-login + args: ["hello", "world"] + env: + - name: "VERSION" + value: "v1.0.1" +``` + +The AWS authenticator above example becomes: + +``` +users: +- name: kubernetes-admin + user: + exec: + kind: "authentication.alpha.k8s.io/bearer-token" + command: heptio-authenticator-aws + # CLUSTER_ID and ROLE_ARN should be replaced with actual desired values. + args: ["token", "-i", "(CLUSTER_ID)", "-r", "(ROLE_ARN)"] +``` + +## TLS client certificate support + +TLS client certificate support is orthogonal to bearer tokens, but something that +we should consider supporting in the future. Beyond requiring different command +output, it also requires changes to the client-go `AuthProvider` interface. + +The current The auth provider interface doesn't let the user modify the dialer, +only wrap the transport. + +``` +type AuthProvider interface { + // WrapTransport allows the plugin to create a modified RoundTripper that + // attaches authorization headers (or other info) to requests. + WrapTransport(http.RoundTripper) http.RoundTripper + // Login allows the plugin to initialize its configuration. It must not + // require direct user interaction. + Login() error +} +``` + +Since this doesn't let a `AuthProvider` supply things like client certificates, +the signature of the `AuthProvider` should change too ([with corresponding changes +to `k8s.io/client-go-transport`]( +https://gist.github.com/ericchiang/7f5804403b359ebdf79dcf76c4071bff)): + +``` +import ( + "k8s.io/client-go/transport" + // ... +) + +type AuthProvider interface { + // UpdateTransportConfig updates a config by adding a transport wrapper, + // setting a bearer token (should ignore if one is already set), or adding + // TLS client certificate credentials. + // + // This is called once on transport initialization. Providers that need to + // rotate credentials should use Config.WrapTransport to dynamically update + // credentials. + UpdateTransportConfig(c *transport.Config) + + // Login() dropped, it was never used. +} +``` + +This would let auth transports supply TLS credentials, as well as instrument +transports with in-memory rotation code like the utilities implemented by +[`k8s.io/client-go/util/certificate`](https://godoc.org/k8s.io/client-go/util/certificate). + +If the exec command specifies `kind: tls-certificate`, the command is expected to +output a unencrypted PEM encoded private key, followed by the PEM encoded client +certificate chain. + +``` +-----BEGIN RSA PRIVATE KEY----- +MIIEowIBAAKCAQEA6IVXGPX5yP2Q6TAlQXIQsavzSqZ973iZvpQBGTI6M98gTSVm +# ... +TUwpZmylTKEJ9zLt2PADglyDrQ2D+1WNzh966Oo9c+kZt4WJM0aF +-----END RSA PRIVATE KEY----- +-----BEGIN CERTIFICATE----- +MIIDCzCCAfOgAwIBAgIJAKK9m2Cfg5uhMA0GCSqGSIb3DQEBCwUAMBsxGTAXBgNV +# ... +xar2YeJ6mCCzSAPM69DP +-----END CERTIFICATE----- +-----BEGIN CERTIFICATE----- +MIIDCzCCAfOgAwIBAgIJAPqJyUfmRxGLMA0GCSqGSIb3DQEBCwUAMBsxGTAXBgNV +# ... +zgmWLORg+ls1H1oaJiNW +-----END CERTIFICATE----- +``` + +The `AuthProvider` then adds those credentials to the `transport.Config`. + +## Login + +Historically, `AuthProviders` have had a `Login()` method with the hope that it +could trigger bootstrapping into the cluster. While no providers implement this +method, the Azure `AuthProvider` can already prompt an [interactive auth flow]( +https://github.com/kubernetes/client-go/blob/kubernetes-1.8.5/plugin/pkg/client/auth/azure/azure.go#L343). +This suggests that an exec'd tool should be able to trigger its own custom logins, +either by opening a browser, or performing a text based prompt. + +We should take care that interactive stderr and stdin are correctly inherited by +the sub-process to enable this kind of interaction. The plugin will still be +responsible for prompting the user, receiving user feedback, and timeouts. + +## Open issues + +### Request specific input + +Credential strategies like [request signing](https://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html) +require client-go to pass the request body or a hash of the body to a plugin. +Additionally, some auth provides react to 401 HTTP status codes (indicating +invalid credentials) to detect misconfigurations or credential expiry. + +As this proposal stands, stdin is used by [login](#login) implementations to receive +user input, while command arguments and environment variables are supplied by the +[exec config](#kubeconfig-changes). It's not clear how we'd pass any more information +to the plugin itself. + +A workaround could be to add hook fields to the `exec` config block. A hypothetical +hook would look like (this document is NOT proposing this exact config change): + +```yaml +users: +- name: kubernetes-admin + user: + exec: + kind: "authentication.alpha.k8s.io/bearer-token" + command: heptio-authenticator-aws + # CLUSTER_ID and ROLE_ARN should be replaced with actual desired values. + args: ["token", "-i", "(CLUSTER_ID)", "-r", "(ROLE_ARN)"] + + # Clear the cache when client-go hits a 401 + on401: + command: heptio-authenticator-aws + args: ["cache", "clear"] +``` + +Another solution would be for client-go to control the environment variables passed +to the plugin: + +``` +KUBERNETES_REQUEST_BODY='{"apiVersion", ... }' +```