Vertical Pod Autoscaler - design proposal. #338
Conversation
k8s-ci-robot
added
the
cncf-cla: yes
|
cc @kubernetes/api-reviewers |
| 9. **History Storage** is a component that consumes utilization signals and OOMs | ||
| (same data as the Recommender) from Heapster and stores it persistently. | ||
| It is used by the Recommender to **initialize its state on startup**. | ||
| First implementation will be InFluxDB. |
There was a problem hiding this comment.
Why InfluxDB? It is now a commercial product and OSS users and so there are no production friendly scalable charts for InfluxDB.
Why not build the initial implementation on top of Prometheus?
There was a problem hiding this comment.
AFAIR Influxdb is currently installed by default on OSS GCE cluster which is not true for Prometheus that requires a bunch of extra steps to be enabled in the first place. So for the initial alpha version we will focus on influxdb.
In the longer run we want to define an API and allow multiple time series databases, including Prometheus, to work with VPA.
There was a problem hiding this comment.
Hrm - prometheus has a simple template that works ootb with kube using the service account token, and already gathers node and pod data. Seems like that would be simpler, but I don't have a strong opinion.
There was a problem hiding this comment.
Here's my proposal: let's try both approaches and see what works. Exploring both solutions will also decrease the risk of biasing the data access API towards one specific implementation.
There was a problem hiding this comment.
We need to focus on something first. I guess we will try influxdb as it is already there and we know how to handle it. But in parallel we may take a look at Prometheus.
In general we are trying to make no extra assumptions about the db apart from that it is possible to put there a metric with a couple metadata and get metrics with metadata from the specified time range.
|
cc @davidopp |
|
|
||
| ### Purpose ### | ||
| Vertical scaling has two objectives: improving utilization of compute resources | ||
| and reducing the maintenance cost, by automating configuration of resource |
There was a problem hiding this comment.
Improving ease of use is the most important objective. Users coming from IaaS aren't used to setting container-level resources. Neither are users coming from a PaaS or FaaS with vertical scaling. And, TBH, we don't have good tools or guidance for people to figure out how to set resource values today. Additionally, I don't want Helm charts or Templates to need to come in tshirt sizes (S/M/L) or to plumb through cpu and memory parameters.
There was a problem hiding this comment.
Totally agree, I put it on the top.
| ### Overview ### | ||
| (see further sections for details and justification) | ||
|
|
||
| 1. We introduce a new type of **API resource**: |
There was a problem hiding this comment.
Did you consider extending LimitRange?
There was a problem hiding this comment.
It could be possible to embed a LimitRange object inside the resourcesPolicy section of the VPA object to express user constraints.
One missing thing in LimitRange however is the ability to define the range for a specific container (identified by name). VPA user may want to define different ranges for different containers in the pod.
There was a problem hiding this comment.
How would we prevent limit range from fighting with VPA? Does VPA have to be earlier in the admission chain than the LimitRange?
| VPA is intended to supersede this feature. | ||
|
|
||
| #### In-place updates #### | ||
| In-place Pod updates is a planned feature to allow changing resources |
| @@ -0,0 +1,486 @@ | |||
| Vertical Pod Autoscaler | |||
| ======================= | |||
| Vertical Pod Autoscaler (VPA, aka. "rightsizing" or "autopilot") is an | |||
There was a problem hiding this comment.
The main issue for this is kubernetes/kubernetes#10782
| (request/limit) of existing containers without killing them, assuming sufficient | ||
| free resources available on the node. Vertical Pod Autoscaler will greatly | ||
| benefit from this ability, however it is not considered a blocker for the MVP. | ||
| #### Resource estimation #### |
There was a problem hiding this comment.
I disagree that this is the difference between RE and VPA.
For others, resource estimation is a concept that comes from Borg:
https://research.google.com/pubs/pub43438.html
I request that we not use the term "resource estimation" in Kubernetes.
We could do node-level vertical pod autoscaling.
Determining the amount of resources available on each node for best-effort workloads is a different problem.
There was a problem hiding this comment.
We could do node-level vertical pod autoscaling
Do you mean letting nodes vertically scale pods directly? If so, I like that idea. It eliminates update latency and it will open up a few more optimizations in the future like that of increasing pod limits before a pod is OOM killed (if the kernel supports necessary primitives).
There was a problem hiding this comment.
From talking with @bgrant0607 my understanding of his proposal is basically three components
- A cluster-level component that
- uses long-term history
- uses global information (all pods of a collection)
- adjusts resource request in pod template for Guaranteed and Burstable pods
- A node-level component that
- uses short-term history (stateless)
- uses only local information (that particular pod on that node)
- adjusts resource request in pod for Burstable and BE pods (relies on in-place update)
- A node-level component that
- uses short-term history (stateless)
- computes some kind of smoothed node-level utilization, when is then used in scheduling best-effort pods (in lieu of capacity minus sum of requests, which is used for scheduling Guaranteed and Burstable pods)
The first thing is a variant of what this design doc is about.
The second thing is a vaiant of what we have called the "resource estimator" in past discussions (e.g. the metrics pipeline diagrams),
The third thing is variant of @piosz's masters project.
By "variant" I don't mean they're the same, I mean there is a rough mapping between the three components @bgrant0607 has in mind and three things that have been talked about in the past.
I agree it is probably a good idea to come up with three different names for these things.
One of the insights @bgrant0607 had was that the second thing does not need to compute some kind of "reservation" value as in Borg; it can just adjust request. (You can't do resource estimation in Borg the same way because Borg conflates "resource consumption at which you get killed" with "resource requirements" so you can't change one without changing the other.)
I think a discussion of whether the first or second component should also be able to change limit (in addition to request) is interesting but opens a can of worms. :)
There was a problem hiding this comment.
In addition to history, the cluster-level component can observe all related pods.
Updating the recommendation in the VPA as in the proposal is roughly equivalent to updating the controller.
There was a problem hiding this comment.
In addition to history, the cluster-level component can observe all related pods.
Yeah, I added "uses global information (all pods of a collection)" shortly after posting the summary; I think that's what you're getting at.
Updating the recommendation in the VPA as in the proposal is roughly equivalent to updating the controller.
Agreed, though if you make the change using an admission controller rather than by modifying the pod template then you can have a different value for each pod of the collection.
There was a problem hiding this comment.
@kgrygiel thoughts on this comment? Given that kubelet can make vertical scaling decisions prior to a container getting OOM killed or running out of storage with low latency, it is beneficial to have the multi-tier model that @bgrant0607 suggested.
| 4. Additionally the Recommender **exposes a synchronous API** that takes a Pod | ||
| description and returns recommended resources. | ||
|
|
||
| 5. All Pod creation requests go through the VPA **Admission Controller**. |
There was a problem hiding this comment.
FYI: the admission control extension proposal:
#132
|
|
||
| Mode can be set to one of the following: | ||
|
|
||
| 1. `"initial"`: VPA only assigns resources on Pod creation and does not |
There was a problem hiding this comment.
FYI, API convention for string constants is CamelCase:
https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md#constants
| rejected. | ||
|
|
||
| ### Life of the Pod ### | ||
| There are two paths for a Pod using VPA to get scheduled. |
There was a problem hiding this comment.
This seems unnecessarily complex for the initial implementation. I'd just use one path.
There was a problem hiding this comment.
Yes, in prototype we will have just one path. The other path will only become possible when Initializers are introduced, so this is a post-prototype extension.
The reason why we designed this in detail so early, was to address possible consequences of VPA pointing at pods and not the other way. We anticipated the problem with race conditions when pod(s) and a corresponding VPA are created together and we wanted to have a story on how this could be addressed in future.
| resource usage patterns, typically a group of replicas or shards of | ||
| a single workload. | ||
|
|
||
| The Recommender exposes a synchronous RPC interface that takes a Pod |
There was a problem hiding this comment.
By "RPC", do you mean grpc?
How will authentication and authorization work?
There was a problem hiding this comment.
Good question. I suppose the canonical way to do this will be the aggregated API servers mechanism (https://github.com/kubernetes/community/blob/master/contributors/design-proposals/aggregated-api-servers.md) ?
There was a problem hiding this comment.
Would end user components be able to hit the recommender api? This could be a "Review" or "Recommendation" API that accepts only POST, but is namespaced:
POST /namespaces/foo/podresourcerecommendations
{
"kind": "PodResourceRecommendation",
"version": "recommender.k8s.io/v1alpha1",
"spec": { pod spec }
"status": {
// set by response
"resources": {
"containers": [{"name": ...}]
}
}
accessed via AA
Will we ever have resources on anything besides containers? If so, we should reserve space in the schema now.
There was a problem hiding this comment.
For now the scope is limited to containers. Yes, we want users to be able to check recommendations, assuming that the configured access policy allows that.
There was a problem hiding this comment.
Until now we've generally had a policy (although not always followed) that all communication between Kubernetes components goes through the API server. I've never really supported this as a blanket policy (e.g. see #6363), but nonetheless that has been the policy and we've considered things like how Heapster works to be exceptions that will eventually be aligned with the policy. Is this design a sign that the policy is not a hard-and-fast rule? (Which would be fine with me.)
There was a problem hiding this comment.
My understanding is that Heapster is an exception only because Aggregated API Servers don't yet exist. Once they are available, Heapster will use this. And VPA recommendations API will be no different.
There was a problem hiding this comment.
Ah OK, I didn't catch the part about using aggregate API servers. Maybe you could make that clearer in the doc (it sounded like direct RPC between components).
| ----------------------- | ||
|
|
||
| ### Pods point at VPA ### | ||
| *REJECTED BECAUSE IT REQUIRES MODIFYING THE POD SPEC* |
There was a problem hiding this comment.
Yes, I'm not a fan of this approach. HPA, LimitRange, and other higher-level automation needs to be loosely coupled from the core system primitives.
|
|
||
| #### cons: #### | ||
| * Extending VPA support from Deployment to other abstractions that manage Pods | ||
| requires additional work. VPA must be aware of all such abstractions. |
There was a problem hiding this comment.
I'm not suggesting we do this, but...
Re. needing to be aware of all such abstractions: We could extend the /scale subresource to include compute resources, or add another polymorphic subresource for that. We could also use the standalone PodTemplate API (kubernetes/kubernetes#170), which would make lots of tooling simpler and would facilitate third-party workload controllers.
| * Extending VPA support from Deployment to other abstractions that manage Pods | ||
| requires additional work. VPA must be aware of all such abstractions. | ||
| * It is not possible to do a rolling update of the VPA config. | ||
| For example setting `max_memory` in the VPA config will apply to the whole |
There was a problem hiding this comment.
I could imagine supporting in-place rolling updates eventually, though probably not soon:
kubernetes/kubernetes#9043
There was a problem hiding this comment.
The problem with this variant is that there is always only one VPA config pointing at the whole Deployment. Given that the config is not a per-pod property, it seems impossible to express the fact that some of the pods use the "old" version of the config and some use the "new" one, which is necessary for a rolling update.
There was a problem hiding this comment.
The VPA could probably take as input on its spec a label with which to subdivide pods under its selector. For deployment, that would be the generation/hash label. For stateful sets, that could be region.
| command, args). | ||
| The recommendation model (MVP) assumes that the memory and CPU consumption are | ||
| independent random variables with distribution equal to the one observed in the | ||
| last N days (recommended value is N=8 to capture weekly peaks). |
There was a problem hiding this comment.
What happens during the first 8 days?
There was a problem hiding this comment.
If the cluster has less than 8 days of recorded history, VPA will use whatever was collected so far.
More geneally, if the recommender has insufficent historical data to give a recommendation for a given VPA it will probably look (as we do internally) at "similar" pods running now or in the past, with a reasonable definition of "similar". Details of the algorithm tbd later, once we have agreement on the overall design.
| 8 days of data. This data is only used to initialize the Recommender on startup. | ||
|
|
||
| ### Open questions ### | ||
| 1. How to resolve conflicts if multiple VPA objects match a Pod. |
There was a problem hiding this comment.
We should use approaches similar to what we've done for ReplicationController in the past. (Someone else can fill in the details.)
There was a problem hiding this comment.
Pod metadata has the ownerReferences list (https://kubernetes.io/docs/api-reference/v1/definitions/#_v1_ownerreference). A references on this list can be marked as "controller" with a promise that there will be no more than one controller.
However we cannot directly reuse this mechanism as is. Presumably we should allow different agents to control different aspects of the pod - we would have to extend this mechanism to allow different types of controllers. Not sure if this is the right way.
Of course this still wouldn't prevent the user from creating a pod that is matched by 2 VPAs. It would merely allow us to choose one of the two as the "active" one for the given pod.
There was a problem hiding this comment.
how do HPA and VPA interact?
There was a problem hiding this comment.
In general HPA and VPA should not point at the same metrics, especially not at the CPU. Simplifying a bit, HPA calculates the number of replicas as
ceil(sum_of_cpu_usage / (cpu_request * target_percentage))
If VPA changes cpu_request then obviously changes the number of replicas and thus changes the CPU load, leading to quite severe oscillations.
We can imagine a case when CPU-based HPA works relatively well with VPA limited only to memory, although it may also have some tricky cases. Anyway we will add a section to the design doc that describes this topic.
| 7. **VPA Updater** is a component responsible for **real-time updates** of Pods. | ||
| If a Pod uses VPA in `"Auto"` mode, the Updater can decide to update it with | ||
| recommender resources. | ||
| In MVP this is realized by just evicting the Pod in order to have it |
There was a problem hiding this comment.
In the MVP, we should probably only evict the pod if it is listed as being owned by a controller. That way if an end user creates a debug pod with the same labels (but without the controller ref), the VPA won't evict their debug / monitoring / trouble shooting pod.
There was a problem hiding this comment.
I'm not sure whether we should allow user to shoot himself into foot or not if he really wants to. VPA acts only on pods selected via a label selector. So if the user points VPA in the wrong direction, bad things may happen. For MVP we can probably also ignore the problem.
There was a problem hiding this comment.
I don't want VPA to be disruptive - if VPA requires deletion, then I don't want it loose in the cluster deleting stuff, I want it to delete only things that are safe to delete. A VPA deleting pods that aren't under a controller is very annoying.
There was a problem hiding this comment.
Ok, we can add a check and publish events when VPA points to a pod that is not backed up by a controller.
kgrygiel
force-pushed
the
master
branch
2 times, most recently
from
fce5815 to
ba546f9
Compare
|
|
||
| // PodResourcePolicy controls how autoscaler computes the recommended resources | ||
| // for containers belonging to the pod. | ||
| type PodResourcePolicy struct { |
There was a problem hiding this comment.
Add information what will happen to containers that are not explicitly mentioned in containers[].
There was a problem hiding this comment.
Strict mode still not added.
There was a problem hiding this comment.
I was thinking about it and frankly I'm a bit reluctant to add this knob before any user even requests it. We would have multiple settings controlling whether VPA is active: UpdateMode + per-container ResourcePolicy + "strict mode". Could we postpone it until we get more clarity whether it is really needed and in what form? We can also discuss more offline.
There was a problem hiding this comment.
I agree with @kgrygiel on this point, KISS rules. Let's figure out what users need first. I suppose it's OK to mention potential policies in the design doc and maybe encourage users to provide feedback based on their workloads/usage but yeah, let's remove this for now.
There was a problem hiding this comment.
What I ask for is not an extra shiny feature but a way for user to express what to do with containers for which min/max memory/cpu is not provided.
Should it be unbounded or not autoscaled at all? Currently there is no convenient way to tell VPA to scale just one container in a Pod. For example scale my web server but leave metrics pusher alone, or any extra container the user may add in the future.
There was a problem hiding this comment.
OK, that sounds sensible @mwielgus and thanks for clarifying it! Why not, rather than making it configurable, choose a default behaviour for now (maybe non-auto-scaling, but no strong opinion here) and gather feedback from users as discussed above?
There was a problem hiding this comment.
Yes, if we go for no-autoscaling by default then we can skip the field for now.
There was a problem hiding this comment.
I'm reluctant to making it no-autoscaling by default, I would strive for the minimal working configuration being really minimal.
After further discussion we agreed to add an (optional) default ContainerResourcePolicy, which addresses the above concerns, while not complicating the API too much, I think.
| // MinRecommended is the recommended minimum amount of resources. | ||
| MinRecommended api.ResourceRequirements | ||
| // MaxRecommended is the recommended maximum amount of resources. | ||
| MaxRecommended api.ResourceRequirements |
There was a problem hiding this comment.
Add some type of confidence information.
|
|
||
| // RecommendationQuerySpec is a request of recommendation for a pod. | ||
| type RecommendationQuerySpec struct { | ||
| PodMetadata metav1.ObjectMeta |
There was a problem hiding this comment.
Let's send the pod struct itself. Will be cleaner.
| Vertical Pod Autoscaler will greatly benefit from this ability, however it is | ||
| not considered a blocker for the MVP. | ||
|
|
||
| #### Resource estimation #### |
There was a problem hiding this comment.
Any reference link?
There was a problem hiding this comment.
Unfortunately I'm not aware of any existing documentation.
| 2. VPA is capable of adjusting container resources of existing Pods, in | ||
| particular reacting to CPU starvation and container OOM events. | ||
|
|
||
| 3. When VPA restarts Pods it respects the disruption budget. |
| (https://github.com/kubernetes/kubernetes/issues/5774)) is a planned feature to | ||
| allow changing resources (request/limit) of existing containers without killing them, assuming sufficient free resources available on the node. | ||
| Vertical Pod Autoscaler will greatly benefit from this ability, however it is | ||
| not considered a blocker for the MVP. |
There was a problem hiding this comment.
minimum viable product
|
Some comments are still not addressed. Please also squash all the commits to just one. |
|
@mwielgus: Can you select "Squash and merge" option while merging? |
| Name string | ||
| // Whether autoscaler is enabled for the container. Defaults to "On". | ||
| // +optional | ||
| ContainerScalingMode |
| user tune the recommendation algorithm to their specific use-case. | ||
|
|
||
| #### Recommendation #### | ||
| The VPA resource has an output-only field keeping a recent recommendation, |
There was a problem hiding this comment.
What are the scale implications of writing this (i.e. if we have 1k VPA, how often do we write this?)
There was a problem hiding this comment.
It may be written every 5-10 minutes (if the recommended resources change), which I suppose shouldn't be an issue?
The freshness of this data is not so critical: both the Admission Controller and the Updater always obtain the most recent recommendation directly from the Recommender API, before updating the pod. Only if the Recommender is down, they may use the recommendation cached in the VPA object.
The only concern is fast reaction to OOMs. The updater should be able to detect OOM-ing containers and call the Recommender API more frequently to ensure fast reaction time.
| type VerticalPodAutoscalerSpec { | ||
| // A label query that determines the set of pods controlled by the Autoscaler. | ||
| // More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors | ||
| Selector *metav1.LabelSelector |
There was a problem hiding this comment.
In this design, a poor / incorrect recommendation can completely break a set of pods (by setting too low limits) and thus for most standard kube apps would take the service offline (most people won't get clever with selectors and blue-green setups). I think this should be called out as a con of the current method below.
There was a problem hiding this comment.
This is something we try to address with the Disruption Budget. The Updater can also have additional heuristics to not take down pods if more than x% of the group is unhealthy.
Is there anything else that can be done? How is it related to the API design?
|
/lgtm |
k8s-ci-robot
added
the
lgtm
|
Remember to squash this before merge... |
k8s-github-robot
removed
the
lgtm
|
/lgtm |
k8s-ci-robot
added
the
lgtm
|
/approve no-issue |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: kgrygiel, mwielgus, wojtek-t The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these OWNERS Files:
You can indicate your approval by writing |
k8s-ci-robot
added
the
approved
|
/assign wojtek-t |
Vertical Pod Autoscaler - design proposal.
Signed-off-by: knight42 <[email protected]>
No description provided.