Heapster nanny container fails to start with Kubernetes 1.9.3

[adelina-t]

ISSUE:

---

v0.13.1:

---

Kubernetes 1.9.3

What happened:

Cluster deployed with Kubernetes 1.9.3 has failed heapster pod.

NAMESPACE     NAME                                            READY     STATUS             RESTARTS   AGE
kube-system   heapster-7cd8dd888b-4556z                       1/2       CrashLoopBackOff   821        2d

How to reproduce it (as minimally and precisely as possible):

Deploy basic cluster from template with orchestrator Kubernetes 1.9.3

Anything else we need to know:

Logs from inside the heapster nanny pod:

kubectl logs heapster-7cd8dd888b-4556z -n kube-system heapster-nanny
ERROR: logging before flag.Parse: I0312 09:56:19.299122       1 pod_nanny.go:64] Invoked by [/pod_nanny --cpu=80m --extra-cpu=0.5m --memory=140Mi --extra-memory=4Mi --threshold=5 --deployment=heapster --container=heapster --poll-period=300000 --estimator=exponential]
ERROR: logging before flag.Parse: I0312 09:56:19.299261       1 pod_nanny.go:76] Watching namespace: kube-system, pod: heapster-7cd8dd888b-4556z, container: heapster.
ERROR: logging before flag.Parse: I0312 09:56:19.299274       1 pod_nanny.go:77] storage: MISSING, extra_storage: 0Gi
ERROR: logging before flag.Parse: I0312 09:56:19.395294       1 pod_nanny.go:162] Failed to read data from config file "MISSING/NannyConfiguration": open MISSING/NannyConfiguration: no such file or directory, using default parameters
ERROR: logging before flag.Parse: I0312 09:56:19.395356       1 pod_nanny.go:166]
ERROR: logging before flag.Parse: I0312 09:56:19.395394       1 pod_nanny.go:101] cpu: , extra_cpu: , memory: , extra_memory:
panic: cannot parse '': quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'

goroutine 1 [running]:
k8s.io/autoscaler/addon-resizer/vendor/k8s.io/apimachinery/pkg/api/resource.MustParse(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/google/home/kawych/go-tools/src/k8s.io/autoscaler/addon-resizer/vendor/k8s.io/apimachinery/pkg/api/resource/quantity.go:136 +0x253
main.main()
	/usr/local/google/home/kawych/go-tools/src/k8s.io/autoscaler/addon-resizer/nanny/main/pod_nanny.go:108 +0x6d5

[feiskyer]

@adelina-t Which image version are you using with heapster-nanny? e.g. you could get by

kubectl -n kube-system get pod heapster-7cd8dd888b-4556z  -o yaml

[adelina-t]

@feiskyer Seems like it's: k8s-gcrio.azureedge.net/addon-resizer:1.8.1

Full pod spec:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/critical-pod: ""
  creationTimestamp: 2018-03-13T10:59:19Z
  generateName: heapster-7cd8dd888b-
  labels:
    k8s-app: heapster
    pod-template-hash: "3784884446"
  name: heapster-7cd8dd888b-shs82
  namespace: kube-system
  ownerReferences:
  - apiVersion: extensions/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: heapster-7cd8dd888b
    uid: 93b74a55-26ad-11e8-a479-000d3af71d5e
  resourceVersion: "2060"
  selfLink: /api/v1/namespaces/kube-system/pods/heapster-7cd8dd888b-shs82
  uid: 949e5a7a-26ad-11e8-a479-000d3af71d5e
spec:
  containers:
  - command:
    - /heapster
    - --source=kubernetes.summary_api:""
    image: k8s-gcrio.azureedge.net/heapster-amd64:v1.5.1
    imagePullPolicy: Always
    name: heapster
    resources:
      limits:
        cpu: 80m
        memory: 140Mi
      requests:
        cpu: 80m
        memory: 140Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: heapster-token-ss7f7
      readOnly: true
  - command:
    - /pod_nanny
    - --cpu=80m
    - --extra-cpu=0.5m
    - --memory=140Mi
    - --extra-memory=4Mi
    - --threshold=5
    - --deployment=heapster
    - --container=heapster
    - --poll-period=300000
    - --estimator=exponential
    env:
    - name: MY_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: MY_POD_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    image: k8s-gcrio.azureedge.net/addon-resizer:1.8.1
    imagePullPolicy: Always
    name: heapster-nanny
    resources:
      limits:
        cpu: 50m
        memory: 90Mi
      requests:
        cpu: 50m
        memory: 90Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: heapster-token-ss7f7
      readOnly: true
  dnsPolicy: ClusterFirst
  nodeName: k8s-master-23902373-0
  nodeSelector:
    beta.kubernetes.io/os: linux
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: heapster
  serviceAccountName: heapster
  terminationGracePeriodSeconds: 30
  tolerations:
  - key: CriticalAddonsOnly
    operator: Exists
  volumes:
  - name: heapster-token-ss7f7
    secret:
      defaultMode: 420
      secretName: heapster-token-ss7f7

[feiskyer]

@adelina-t Current config only applies to add-resizer v1.7, e.g. replace image with k8s-gcrio.azureedge.net/addon-resizer:1.8.1.

Or if addon-resizer 1.8.1 is required,

apiVersion: v1
kind: ServiceAccount
metadata:
  name: heapster
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: heapster-config
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: EnsureExists
data:
  NannyConfiguration: |-
    apiVersion: nannyconfig/v1alpha1
    kind: NannyConfiguration
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: heapster
  namespace: kube-system
  labels:
    k8s-app: heapster
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    addonmanager.kubernetes.io/mode: EnsureExists
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: heapster
  template:
    metadata:
      labels:
        k8s-app: heapster
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      priorityClassName: system-cluster-critical
      containers:
        - image: k8s.gcr.io/heapster-amd64:v1.5.0
          name: heapster
          resources:
            limits:
              cpu: 88m
              memory: 204Mi
            requests:
              cpu: 88m
              memory: 204Mi
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8082
              scheme: HTTP
            initialDelaySeconds: 180
            timeoutSeconds: 5
          command:
            - /heapster
            - --source=kubernetes.summary_api:''
        - image: k8s.gcr.io/addon-resizer:1.8.1
          name: heapster-nanny
          resources:
            limits:
              cpu: 88m
              memory: 204Mi
            requests:
              cpu: 88m
              memory: 204Mi
          env:
            - name: MY_POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: MY_POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          volumeMounts:
          - name: heapster-config-volume
            mountPath: /etc/config
          command:
            - /pod_nanny
            - --config-dir=/etc/config
            - --cpu=80m
            - --extra-cpu=0.5m
            - --memory=140Mi
            - --extra-memory=4Mi
            - --threshold=5
            - --deployment=heapster
            - --container=heapster
            - --poll-period=300000
            - --estimator=exponential
      volumes:
        - name: heapster-config-volume
          configMap:
            name: heapster-config
      serviceAccountName: heapster
      tolerations:
        - key: "CriticalAddonsOnly"
          operator: "Exists"

[feiskyer]

@jackfrancis I see addon-resizer version has been rolled back in #2090. You can refer above manifests to fix addon-resizer:1.8.1

[jackfrancis]

@feiskyer I interpret your comments as a suggestion to add a new version of this spec to pair w/ addon-resizer v1.8.1:

https://github.com/Azure/acs-engine/blob/master/parts/k8s/addons/kubernetesmasteraddons-heapster-deployment.yaml

Which versions of Kubernetes do you recommend that we deliver a modified heapster addon spec + addon-resizer v1.8.1 with?

Thanks so much!

[feiskyer]

@jackfrancis Yep, I suggest we add addon-resizer:v1.8.1 with kubernetes v1.9.x and above versions. Addons in Kubernetes repo are also using resizer v1.8.1 from v1.9.x

[PatrickLang]

Looks like 1.9.6 works