Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VPA admission controller should stop review Pod Update event #1152

Merged
merged 1 commit into from
Aug 20, 2018
Merged

VPA admission controller should stop review Pod Update event #1152

merged 1 commit into from
Aug 20, 2018

Conversation

phsiao
Copy link
Contributor

@phsiao phsiao commented Aug 15, 2018

I might be wrong, but it is my understanding that pod Update review has no impact through VPA admission controller. It would be doing work that will have no effect at the end.

There can be a lot of pod updates events, for example, updating pod annotations, and would be creating a lot of unnecessary work. I'd like to propose we remove the review of pod Update.

@k8s-ci-robot k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Aug 15, 2018
@phsiao
Copy link
Contributor Author

phsiao commented Aug 15, 2018

/cc @bskiba

@k8s-ci-robot k8s-ci-robot requested a review from bskiba August 15, 2018 22:11
@bskiba
Copy link
Member

bskiba commented Aug 20, 2018

It seems reasonable (and desired) since there should not be changes to resources in the updates (at least in the current k8s version). However, I'd like to see this tested, to make sure the changes introduced by VPA admission controller do not get overriden, for example if there are resource request specified in the deployment template. Could you perform such tests and describe outcome here?

@phsiao
Copy link
Contributor Author

phsiao commented Aug 20, 2018

Sure, happy to verify that. Here is my understanding of what should happen, but I will verify that.

  1. a workload object is updated with different resource request
  2. the workload controller should start replacing the pods by creating new pods and terminating old pods
  3. the vpa admission controller should still see all the Create event for the new pods, and patch the resource request accordingly.

@bskiba
Copy link
Member

bskiba commented Aug 20, 2018

That's my understanding too. Thanks for taking the time to make sure!

@phsiao
Copy link
Contributor Author

phsiao commented Aug 20, 2018

Here is what I did and attempt to verify:

  1. Deploy this hamster vpa and deployment,
apiVersion: "poc.autoscaling.k8s.io/v1alpha1"
kind: VerticalPodAutoscaler
metadata:
  name: hamster-vpa
  namespace: sandbox
spec:
  selector:
    matchLabels:
      app: hamster
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: hamster
  namespace: sandbox
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: hamster
    spec:            
      containers:
      - name: hamster
        image: k8s.gcr.io/ubuntu-slim:0.1
        resources:
          requests:
            cpu: 100m
            memory: 50Mi
        command: ["/bin/sh"]
        args: ["-c", "while true; do timeout 0.5s yes >/dev/null; sleep 0.5s; done"]
  1. let it run for a while and get a snapshot of vpa
apiVersion: poc.autoscaling.k8s.io/v1alpha1
kind: VerticalPodAutoscaler
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"poc.autoscaling.k8s.io/v1alpha1","kind":"VerticalPodAutoscaler","metadata":{"annotations":{},"name":"hamster-vpa","namespace":"sandbox"},"spec":{"selector":{"matchLabels":{"app":"hamster"}}}}
  clusterName: ""
  creationTimestamp: 2018-08-20T16:28:08Z
  generation: 0
  name: hamster-vpa
  namespace: sandbox
  resourceVersion: "770208756"
  selfLink: /apis/poc.autoscaling.k8s.io/v1alpha1/namespaces/sandbox/verticalpodautoscalers/hamster-vpa
  uid: 05bd24dd-a496-11e8-ae2b-90b11c5787b7
spec:
  selector:
    matchLabels:
      app: hamster
  updatePolicy:
    updateMode: Auto
status:
  conditions:
  - lastTransitionTime: 2018-08-20T16:31:47Z
    status: "True"
    type: RecommendationProvided
  recommendation:
    containerRecommendations:
    - containerName: hamster
      lowerBound:
        cpu: 555m
        memory: "104857600"
      target:
        cpu: 564m
        memory: "104857600"
      upperBound:
        cpu: 4884m
        memory: "151867269"
  1. delete one of the hamster pod and let it re-create with the vpa admission controller, verify the new pod is created with the recommended request
    resources:
      requests:
        cpu: 564m
        memory: "104857600"
  1. increase the request in the yaml in Fix imports in cluster autoscaler after migrating it from contrib #1 to be 1000m cpu and 500Mi memory, apply to the cluster, observe the two new pods have
    resources:
      requests:
        cpu: 564m
        memory: "104857600"

and

    resources:
      requests:
        cpu: 564m
        memory: "104857600"

(identical for both new pods)


Note that this is running with patched admission controller that only observes CREATE event for pod.

  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods
  - apiGroups:
    - poc.autoscaling.k8s.io
    apiVersions:
    - v1alpha1
    operations:
    - CREATE
    - UPDATE
    resources:
    - verticalpodautoscalers

@phsiao
Copy link
Contributor Author

phsiao commented Aug 20, 2018

@bskiba please see above for my attempt to verify that. I don't know what we can do to put Update back when patching resource request in-place is supported, but I think that is going to be a pretty obvious bug when that happens.

@bskiba
Copy link
Member

bskiba commented Aug 20, 2018

Thanks! Looks good. We'll be watching pretty closely for the in-place updates of resource requests as we are very keen to use them for our own advantage :)
/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 20, 2018
@bskiba bskiba merged commit 7857102 into kubernetes:master Aug 20, 2018
@phsiao phsiao deleted the stop_review_pod_update branch August 20, 2018 18:56
yaroslava-serdiuk pushed a commit to yaroslava-serdiuk/autoscaler that referenced this pull request Feb 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/vertical-pod-autoscaler cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants