Add comprehensive horizontal pod autoscaling documentation

docs/cluster_spec.md

@@ -275,7 +275,7 @@ spec:
     horizontalPodAutoscalerUpscaleDelay: 3m0s
 ```
 
-For more details on `horizontalPodAutoscaler` flags see the [HPA docs](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/).
+For more details on `horizontalPodAutoscaler` flags see the [official HPA docs](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) and the [Kops guides on how to set it up](horizontal_pod_autoscaling.md).
 
 ####  Feature Gates
 

docs/horizontal_pod_autoscaling.md

@@ -0,0 +1,133 @@
+# Horizontal Pod Autoscaling
+
+With Horizontal Pod Autoscaling, Kubernetes automatically scales the number of
+pods in a replication controller, deployment or replica set based on observed
+CPU utilization (or, with alpha support, on some other, application-provided
+metrics).
+
+The current stable version, which only includes support for CPU autoscaling, can
+be found in the `autoscaling/v1` API version. The alpha version, which includes
+support for scaling on memory and custom metrics, can be found in
+`autoscaling/v2alpha1` in 1.7 and `autoscaling/v2beta1` 1.8 and 1.9.
+
+Kops can assist in setting up HPA and recommends Kubernetes `1.7.x` to `1.9.x`
+and Kops `>=1.7`. Relevant reading you will need to go through:
+
+* [Extending the Kubernetes API with the aggregation layer][k8s-extend-api]
+* [Configure The Aggregation Layer][k8s-aggregation-layer]
+* [Horizontal Pod Autoscaling][k8s-hpa]
+
+While the above links go into details on how Kubernetes needs to be configured
+to work with HPA, a lot of that work is already done for you by Kops.
+Specifically:
+
+* [x] Enable the [Aggregation Layer][k8s-aggregation-layer] via the following
+  kube-apiserver flags:
+   * [x] `--requestheader-client-ca-file=<path to aggregator CA cert>`
+   * [x] `--requestheader-allowed-names=aggregator`
+   * [x] `--requestheader-extra-headers-prefix=X-Remote-Extra-`
+   * [x] `--requestheader-group-headers=X-Remote-Group`
+   * [x] `--requestheader-username-headers=X-Remote-User`
+   * [x] `--proxy-client-cert-file=<path to aggregator proxy cert>`
+   * [x] `--proxy-client-key-file=<path to aggregator proxy key>`
+* [x] Enable [Horizontal Pod Scaling][k8s-hpa] ... set the appropriate flags for
+  `kube-controller-manager`:
+   * [x] `--horizontal-pod-autoscaler-use-rest-clients` should be true.
+   * [x] `--kubeconfig <path-to-kubeconfig>`
+
+Ensure that you use **1.8.5 at minimum for Kubernetes 1.8.x** due to [a nasty
+bug affecting the API aggregation layer][issue-1]. It was fixed in
+[kubernetes/kubernetes#55259][pr-5].
+
+## Cluster Configuration
+
+### Support For Multiple Metrics
+
+Enable API versions required to support scaling on cpu, memory and custom
+metrics:
+
+```yaml
+# On K8s 1.7
+spec:
+  kubeAPIServer:
+    runtimeConfig:
+      autoscaling/v2alpha1: "true"
+```
+
+```yaml
+# On K8s 1.8 and 1.9
+spec:
+  kubeAPIServer:
+    runtimeConfig:
+      autoscaling/v2beta1: "true"
+```
+
+If you've set the above configuration, your cluster is now ready for the
+resource metrics API ([installation instruction here][k8s-metrics-server]). The
+compatibility matrix is as follows:
+
+Metrics Server | Metrics API group/version | Supported Kubernetes version
+---------------|---------------------------|-----------------------------
+0.2.x          | `metrics.k8s.io/v1beta1`  | 1.8+
+0.1.x          | `metrics/v1alpha1`        | 1.7
+
+### Support For Custom Metrics
+
+Enable gathering custom metrics:
+
+```yaml
+spec:
+  kubelet:
+    enableCustomMetrics: true
+```
+
+Enable the horizontal pod autoscaler REST client:
+
+```yaml
+spec:
+  kubeControllerManager:
+    horizontalPodAutoscalerUseRestClients: true
+```
+
+If you've set the above configuration, your cluster is now ready for the custom
+metrics API. Register it via the API aggregation layer. If you're using
+Prometheus, checkout the [custom metrics adapter for Prometheus][k8s-prometheus-custom-metrics-adapter].
+
+## Implementation Details
+
+These are the PRs that enable the required configuration:
+
+* [kubernetes/kops#3679][pr-1] - sets `--requestheader-xxx` kube-apiserver flags
+  required to enable aggregation layer
+  ```
+  --requestheader-client-ca-file=<path to aggregator CA cert>
+  --requestheader-allowed-names=aggregator
+  --requestheader-extra-headers-prefix=X-Remote-Extra-
+  --requestheader-group-headers=X-Remote-Group
+  --requestheader-username-headers=X-Remote-User
+  ```
+* [kubernetes/kops#3165][pr-2] - sets `--proxy-client-xxx` kube-apiserver flags
+  required to enable aggregation layer
+  ```
+  --proxy-client-cert-file=<path to aggregator proxy cert>
+  --proxy-client-key-file=<path to aggregator proxy key>
+  ```
+* [kubernetes/kops#3939][pr-3] - add config option to set `--horizontal-pod-
+  autoscaler-use-rest-clients` kube-controller-manager flag required to enable
+  custom metrics
+* [kubernetes/kops#1574][pr-4] - add config options to set `--enable-custom-
+  metrics` flag on master and node kubelets required to enable custom metrics
+
+[k8s-aggregation-layer]: https://v1-9.docs.kubernetes.io/docs/tasks/access-kubernetes-api/configure-aggregation-layer/
+[k8s-extend-api]: https://v1-9.docs.kubernetes.io/docs/concepts/api-extension/apiserver-aggregation/
+[k8s-hpa]: https://v1-9.docs.kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
+[k8s-metrics-server]: https://github.com/kubernetes-incubator/metrics-server#deployment
+[k8s-prometheus-custom-metrics-adapter]: https://github.com/DirectXMan12/k8s-prometheus-adapter
+
+[pr-1]: https://github.com/kubernetes/kops/pull/3679
+[pr-2]: https://github.com/kubernetes/kops/pull/3165
+[pr-3]: https://github.com/kubernetes/kops/pull/3939
+[pr-4]: https://github.com/kubernetes/kops/pull/1574
+[pr-5]: https://github.com/kubernetes/kubernetes/pull/55259
+
+[issue-1]: https://github.com/kubernetes/kubernetes/issues/55022