Kubernetes 1.2 not reporting RC metadata.labels.* as tags

[albertsun]

We're running agent version 5.7.4 as a DaemonSet in our kubernetes cluster.

Our apps have replication controllers with metadata like this.

"metadata": {
    "name": "intl-v1.288-http",
    "namespace": "adm",
    "labels": {
      "app": "intl",
      "role": "http",
      "version": "v1.288"
    }
  },

In our dashboards we would filter metrics based on those labels. For instance we were getting kube_app and kube_role tags on the metrics. Since upgrading to kubernetes 1.2, those tags no longer appear for apps in our cluster.

This seems related to #2388 or whatever refactoring is happening for 5.8, right?

Will these kube_app and etc labels return in version 5.8? Is there some other option to re-enable them?

[irabinovitch]

@hkaj any thoughts here?

[remh]

Yes the refactoring i'm working on will fix this. Thanks for the feedback @albertsun

[albertsun]

Thanks! Good to hear.

Saw in an old comment on the other issue that 5.8.0 would be released in the next couple of weeks — is there any update on that timeline? Otherwise we have to choose between not upgrading our kubernetes cluster or upgrading and having not-optimal metrics for a while.

[remh]

@albertsun Release should happen next week!

[albertsun]

Cool! So I switched our daemonset to use this docker image datadog/docker-dd-agent:remh_kube1.2 just now to see if these latest changes would work.

Seems that the kube_app label is back now, but we're now missing an environment label which we had set on each of the nodes to filter between prd and stg. It's set as a custom datadog host tag and as an AWS instance tag. Any idea why that has now gone missing?

[remh]

Thanks for the feedback @albertsun !
How did you set the host tag ? Did you pass the environment variable properly to the agent container ?

[albertsun]

The environment tag is set on the worker node instance that the dd-agent
pod runs on. Is there somewhere in the daemonset config it should be set
instead?
On Wed, May 18, 2016 at 5:06 PM Remi Hakim [email protected] wrote:

Thanks for the feedback @albertsun https://github.com/albertsun !
How did you set the host tag ? Did you pass the environment variable
properly to the agent container ?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#2494 (comment)

[remh]

Yes your DS manifest file should look like

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: dd-agent
spec:
  template:
    metadata:
      labels:
        app: dd-agent
      name: dd-agent
    spec:
      containers:
      - image: datadog/docker-dd-agent:remh_kube1.2
        imagePullPolicy: Always
        name: dd-agent
        ports:
          - containerPort: 8125
            name: dogstatsdport
        env:
          - name: API_KEY
            value: $YOUR_API_KEY
          - name: TAGS
            value: "environment:prd"
        volumeMounts:
          - name: dockersocket
            mountPath: /var/run/docker.sock
          - name: procdir
            mountPath: /host/proc
            readOnly: true
          - name: cgroups
            mountPath: /host/sys/fs/cgroup
            readOnly: true
      volumes:
        - hostPath:
            path: /var/run/docker.sock
          name: dockersocket
        - hostPath:
            path: /proc
          name: procdir
        - hostPath:
            path: /sys/fs/cgroup
          name: cgroups

(See the env section)

Also please note that the remh_kube1.2 tag is a working tag and might disappear at any time.

[albertsun]

Great thanks! Added that to the env and now all the right tags seem to be showing up.

[albertsun]

Just checking in on this ticket, I've still been running the remh_kube1.2 build. Has this been merged into the datadog/docker-dd-agent:kubernetes image yet? Just want to know when to switch back.

[remh]

@albertsun Yes it has. I'm closing this issue.

[albertsun]

Thanks!