USHIFT-2188: introduce microshift API Custom certs

Signed-off-by: Evgeny Slutsky <[email protected]>

enhancements/microshift/microshift-apiserver-custom-certs.md

@@ -0,0 +1,191 @@
+---
+title: microshift-apiserver-custom-certs
+authors:
+  - "@eslutsky"
+reviewers:
+  - "@jerpeter, MicroShift contributor"
+  - "@pacevedom, MicroShift contributor"
+  - "@ggiguash, MicroShift contributor"
+  - "@benluddy, OpenShift API server team"
+  - "@standa, OpenShift Auth Team"
+
+approvers:
+  - "@jerpeter"
+api-approvers:
+  - N/A
+creation-date: 2021-01-18
+last-updated: 2023-01-18
+tracking-link:
+  - https://issues.redhat.com/browse/USHIFT-2101
+see-also:
+  - microshift-apiserver-certs.md
+replaces:
+  - N/A
+superseded-by:
+  - N/A
+---
+
+# MicroShift API server custom external certificates
+## Summary
+
+This enhancement extends the Microshift apiserver Certs to allow the user to
+configure the use of client certificates for api server external TLS access.
+
+
+> Anytime the document mentions API server, it refers to kube API server.
+
+
+## Motivation
+Currently, users of Microshift cannot provide their own generated certificates,
+Some customers have very strict requirements regarding TLS certs. There are frequently intermediate proxies which allow only TLS traffic with certs that are signed by organization owned CAs.
+
+### User Stories
+* As a MicroShift administrator, I want to be able to add  organization generated certificates.
+* As a MicroShift administrator, I want to provide additional DNS names for each certificate.
+* As a MicroShift administrator, I want to be able to remove organization generated certificates.
+* As a MicroShift administrator, I want to be able to remove DNS names for each certificate.
+
+### Goals
+* Allow MicroShift administrator to configure Microshift with externally generated certificates and
+domain names.
+
+### Non-Goals
+N/A
+
+## Proposal
+Proposal suggests to provide Administrator means adding their own Certificates using microshift configuration file.
+
+a new `apiServer` level section will be added to the configuration called `namedCertificates`:
+
+```yaml
+apiServer:
+  namedCertificates:
+  - certPath: ~/certs/api_fqdn_1.crt
+    keyPath:  ~/certs/api_fqdn_1.key
+  - certPath: ~/certs/api_fqdn_2.crt
+    keyPath:  ~/certs/api_fqdn_2.key
+    names:
+    - api_fqdn_1     
+    - *.apps.external.com
+
+```  
+
+for each provided certificate, The following configuration is proposed:
+1. `names` - optional list of explicit DNS names (leading wildcards allowed).
+   If no names are provided, the implicit names will be extracted from the certificates.
+1. `certPath` -  certificate full path.
+1. `keyPath` -   certificate key full path.
+
+certificate files will be read from their conifgured location by the Microshift service,
+each certification file will be validated (see validation rules).
+
+Those certificate will extend the default  API server [external](https://github.com/openshift/microshift/blob/main/pkg/controllers/kube-apiserver.go#L194) certificate configuration.
+
+
+### Workflow Description
+
+#### Default API Certs
+1. by default when there is no namedCertificates configuration the behaviour will remain the same.
+
+#### when custom namedCertificates configured
+1. Device Administrator copies the certificated to MicroShift host
+1. Device Administrator configures `namedCertificates` in the Microshift configuration yaml file (/etc/microshift/config.yaml).
+1. Device Administrator start/restarts MicroShift
+1. During startup the certifates paths will be checked and validated. in case of an error service will produce clear error log and exit.
+1. Certifates will be passed to the tls-sni-cert-key command line flag preceding all the other certificates.
+1. kube-apiserver picks up the certificates and start serving them on the configured SNI.
+
+### Implementation Details
+
+The certs will be prepended to the []configv1.NamedCertificate list before the api server is started. (it will be added to `-tls-sni-cert-key` flag)
+
+this certification paths configuration and names will be prepended into the kube-apiserver `tls-sni-cert-key` command line flag.
+when same SNI appear in the CN part of the provided certs,this certificate will take precedence over the `default` external-signer. [ref](https://github.com/kubernetes/kubernetes/blob/98358b8ce11b0c1878ae7aa1482668cb7a0b0e23/staging/src/k8s.io/apiserver/pkg/server/dynamiccertificates/named_certificates.go#L38)
+
+
+### API Extensions
+N/A
+
+### Risks and Mitigations
+* User provide incorrect/expired certificate.
+  kube-api server will continue serving with an expired cert - similiar approach is taken by OpenShift.
+  > users can mitigate this by using --insecure-skip-tls-verify client mode
+
+* User provided expired Certificate and Microshift service started/restarted 
+  Microshift will start with a warning in the logs,kube-api will continue serve with an expired cert - similiar approach is taken by OpenShift.
+  > users can mitigate this by using --insecure-skip-tls-verify client mode
+
+### Drawbacks
+* External certificate wont be automaticly renewed, so it requires manual Certificate rotation. 
+  similiar approach is taken by OpenShift.
+
+
+## Design Details
+N/A
+
+### Open Questions [optional]
+- How to handle kubeconfig generation for NamedCerts?
+
+### Test Plan
+add e2e test that will:
+- generate and sign certificates with custom ca.
+- change the default Microshift configuration to use the newly generated certs.
+- make sure system is functional using  generated external kubeconfig.
+
+### Graduation Criteria
+#### Dev Preview -> Tech Preview
+- Gather feedback from early adopters on possible issues.
+
+#### Tech Preview -> GA
+- Extensive testing with various certificates variations.
+
+#### Removing a deprecated feature
+N/A
+
+### Upgrade / Downgrade Strategy
+N/A
+
+### Version Skew Strategy
+N/A
+
+### Operational Aspects of API Extensions
+N/A
+
+#### Failure Modes
+the provided certs value will be validated before is it passed to the api-server flag
+
+this check will prevent Microshift service from starting:
+1. certificates files exists in the disk and readable by microshift process.
+1. certificates shouldnt override the internal local certificates which are managed internally.
+
+this check display warning message at the log and service will be started:
+1. certificates is expired.
+
+
+#### Support Procedures
+configured certs value are prepended to the  `tls-sni-cert-key`  TLS handshake command line flag  which is passed to the kube-apiserver:
+```
+> journalctl -u microshift -b0 | grep tls-sni-cert-key
+
+Jan 24 14:53:00 localhost.localdomain microshift[45313]: kube-apiserver I0124 14:53:00.649099   45313 flags.go:64] FLAG: --tls-sni-cert-key="[/home/eslutsky/dev/certs/server.crt,/home/eslutsky/dev/certs/server.key;/var/lib/microshift/certs/kube-apiserver-external-signer/kube-external-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-external-signer/kube-external-serving/server.key;/var/lib/microshift/certs/kube-apiserver-localhost-signer/kube-apiserver-localhost-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-localhost-signer/kube-apiserver-localhost-serving/server.key;/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.key
+```
+
+Connecting to API server using external SNI yields external certificate.
+   ```shell
+   $ openssl s_client -connect <SNI_ADDRESS>:6443 -showcerts | openssl x509 -text -noout -in - | grep -C 1 "Alternative\|CN"
+   ```
+
+## Implementation History
+N/A
+
+## Alternatives
+- Use cert-manager for managing Microshift external custom certs
+  - Benifits
+    1. will allow Microshift Administrator create automatic Certificate rotation strategy using supported protocols  (ie: ACME Webhook)
+
+  - Drawbacks
+    1. cert-manager is for managing certs inside k8s clusters. (its not designed for kube-api certs which is based on files.)
+    1. cert-managed will introduce additional load (CPU/Memory) into Microshift Node.
+
+## Infrastructure Needed [optional]
+N/A