Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove total resource calculation proposal from the sidecar KEP #4863

Merged
merged 1 commit into from
Sep 19, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 4 additions & 73 deletions keps/sig-node/753-sidecar-containers/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,9 +107,6 @@ tags, and then generate with `hack/update-toc.sh`.
- [With sidecar container feature](#with-sidecar-container-feature)
- [Resources calculation for scheduling and pod admission](#resources-calculation-for-scheduling-and-pod-admission)
- [Exposing Pod Resource requirements](#exposing-pod-resource-requirements)
- [Goals of exposing the Pod.TotalResourcesRequested field](#goals-of-exposing-the-podtotalresourcesrequested-field)
- [Implementation details](#implementation-details)
- [Notes for implementation](#notes-for-implementation)
- [Resources calculation and Pod QoS evaluation](#resources-calculation-and-pod-qos-evaluation)
- [Resource calculation and version skew](#resource-calculation-and-version-skew)
- [Topology and CPU managers](#topology-and-cpu-managers)
Expand Down Expand Up @@ -864,76 +861,10 @@ It is too much to ask of users to perform this even more complex calculation
simply to know the amount of free capacity they need for a given resource to
allow a pod to schedule.

#### Goals of exposing the Pod.TotalResourcesRequested field

- Allow Kubernetes users to quickly identify the effective resource requirements
for a pending or scheduled pod directly via observing the pod status.
- Ability to cluster autoscaler, karpenter etc to collect a bunch of resource
unavailable/unschedulable pods and easily sum up their
`TotalResourcesRequested` to add nodes
- Allow consuming the Pod Requested Resources via metrics:
- Make sure kube_pod_resource_requests formula is up to date
- Consider exposing Pod Requirements as a Pod state metric via
https://github.com/kubernetes/kube-state-metrics/blob/main/docs/pod-metrics.md
- Provide a well documented, reusable, exported function to be used to calculate
the effective resource requirements for a v1.Pod struct.
- Eliminate duplication of the pod resource requirement calculation within
`kubelet` and `kube-scheduler`.

Note: in order to support the [Downgrade strategy](#downgrade-strategy), scheduler
will ignore the presence of the feature gate when calculating resources. This will
prevent overbooking of nodes when scheduler ignored sidecar when calculating resources
and scheduled too many Pods on the Node that had the feature gate enabled.

#### Implementation details

We propose making two changes to satisfy the two primary consumers of the
effective pod resource requests, users and other Kubernetes ecosystem software
components.

The first change is to add a field to `PodStatus` to represent the effective
resource requirements for the pod. The field is named
`TotalResourcesRequested`. This field allows users to inspect running and
pending pods to quickly determine their effective resource requirements. The
field will be first populated by scheduler in the
[updatePod](https://github.com/kubernetes/kubernetes/blob/815e651f397a6f5691efb26b0d4f3bddb034668c/pkg/scheduler/schedule_one.go#L943).

The updating of this field would occur the existing
[generateAPIPodStatus](https://github.com/kubernetes/kubernetes/blob/a668924cb60901b413abc1fe7817bc7969167064/pkg/kubelet/kubelet_pods.go#L1459) method.

```
// TotalResourcesRequested is the effective resource requirements for this pod, taking into consideration init containers, sidecars, containers and pod overhead.
// More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
// +optional
TotalResourcesRequested ResourceList `json:"totalResourcesRequested,omitempty" protobuf:"bytes,8,opt,name=totalResourcesRequested"`
```

The second change is to update the [PodRequestsAndLimitsReuse](https://github.com/kubernetes/kubernetes/blob/dfc9bf0953360201618ad52308ccb34fd8076dff/pkg/api/v1/resource/helpers.go#L64) function to support
the new calculation and, if possible, re-use this functionality the other places
that pod resource requests are needed (e.g. kube-scheduler, kubelet). This
ensures that components within Kubernetes have an identical computation for
effective resource requirements and will reduce maintenance effort. Currently
this function is only used for the metrics `kube_pod_resource_request` and
`kube_pod_resource_limit` that are exposed by `kube-scheduler` which align with the
values that will also now be reported on the pod status.

A correct, exported function is particularly useful for other Kubernetes
ecosystem components that need to know the resource requirements for pods that
don’t exist yet. For example, an autoscaler needs to know what the resource
requirements will be for DaemonSet pods when they are created to incorporate
them into its calculations if it supports scale to zero.

#### Notes for implementation

This change could be made in a phased manner:
- Refactor to use the `PodRequestsAndLimitsReuse` function in all situations where pod resource requests are needed.
- Add the new `TotalResourcesRequested` field on `PodStatus` and modify
`kubelet` & `kube-scheduler` to update the field.

These two changes are independent of the sidecar and in-place resource update
KEPs. The first change doesn’t present any user visible change, and if
implemented, will in a small way reduce the effort for both of those KEPs by
providing a single place to update the pod resource calculation.
This KEP will not expose the total resource requests field to end user
as many decisions on this field need to be made from other KEPs: InPlace pod update
and Pod Level resources. We do not want to make it harder for those new KEPs
to be implemented by exposing this field prematurely.

#### Resources calculation and Pod QoS evaluation

Expand Down