[kubernetes] Use pod annotations for service discovery #2794
Assignees
Labels
Comments
mikekap
added a commit
to mikekap/dd-agent
that referenced
this issue
Sep 15, 2016
…annotations. This change makes the docker service discovery read the kubernetes annotations to discover how to monitor a pod. This behavior is only triggered when a service discovery backend isn't set. The 3 annotations looked for are: - `com.datadoghq.sd/check_names` - `com.datadoghq.sd/init_configs` - `com.datadoghq.sd/instances` The semantics are exactly the same as that of a KV store. Fixes DataDog#2794
mikekap
added a commit
to mikekap/dd-agent
that referenced
this issue
Sep 15, 2016
…annotations. This change makes the docker service discovery read the kubernetes annotations to discover how to monitor a pod. This behavior is only triggered when a service discovery backend isn't set. The 3 annotations looked for are: - `com.datadoghq.sd/check_names` - `com.datadoghq.sd/init_configs` - `com.datadoghq.sd/instances` The semantics are exactly the same as that of a KV store. Fixes DataDog#2794
mikekap
added a commit
to mikekap/dd-agent
that referenced
this issue
Sep 15, 2016
…annotations. This change makes the docker service discovery read the kubernetes annotations to discover how to monitor a pod. This behavior is only triggered when a service discovery backend isn't set. The 3 annotations looked for are: - `com.datadoghq.sd/check_names` - `com.datadoghq.sd/init_configs` - `com.datadoghq.sd/instances` The semantics are exactly the same as that of a KV store. Fixes DataDog#2794
mikekap
added a commit
to mikekap/dd-agent
that referenced
this issue
Sep 16, 2016
…annotations. This change makes the docker service discovery read the kubernetes annotations to discover how to monitor a pod. This behavior is only triggered when a service discovery backend isn't set. The 3 annotations looked for are: - `com.datadoghq.sd/check_names` - `com.datadoghq.sd/init_configs` - `com.datadoghq.sd/instances` The semantics are exactly the same as that of a KV store. Fixes DataDog#2794
|
Is there any documentation or examples for this? What's the purpose of |
|
Annotations apply to pods which can embed several containers. Even with a single container you may want to run several checks against it. For example an nginx container could require the nginx check and an http_check for a specific page it's serving. Sadly the documentation is a bit behind here, we will update it to include explanations and examples for this feature asap. |
|
Can you provide a really trivial example? As far as I can tell, in a vanilla docker setting, |
|
|
|
Oops, yes, my mistake/misinterpretation. Thanks for clarifying! |
|
@mikekap I think I'm doing something wrong, but I cant figure out what. I have a running GKE cluster with a default dd-daemonset. Added LOG_LEVEL=DEBUG as env. First confusing thing is, I get the same output in my dd-pod logs as with INFO: After that I've added a redis pod+service with the following configuration: I can't see any metrics in my dd Redis dashboard. Do you have a clue what I'm doing wrong? |
|
@stvnwrgs You probably want to check the dd-agent collector logs - you'd want to exec into the dd-agent container and Separately, you may want to be careful depending on this (undocumented) feature for now since #2901 changes it a little (the annotations are different). |
|
@mikekap: Thanks for the tip! Do I also have to set SD_CONFIG_BACKEND? From the commits it's not quite clear for me. Edit: I get a lot of: And |
|
I have two environment variables: Other than that, it looks like it should work. |
|
Well there is still nothing to see in my dashboard. Shouldn't there be any logs like: "redis metric collector started, gained matrics , pushed metrics" etc ? Here is a little bit more output from the collector: |
|
One thing of note is the collector doesn't pick up annotation changes when the container doesn't restart (i.e. if you just edit the pod and add annotations, dd-agent won't pick that up). Could you try restarting dd-agent and post the logs from the collector's launch? |
|
I did. Same behavior as before. |
|
Hi @stvnwrgs |
|
@hkaj Opened the ticket, but still no response... https://help.datadoghq.com/hc/en-us/requests/70927 More documentation about this would be really helpful! The docs don't describe kubernetes as backend. |
|
@stvnwrgs in case you or someone else is still running into this, the feature is documented now, but it is not in a released version yet. It should be part of the 5.11 release. I had submitted a support ticket since I was confused by this too. I had just installed the latest agent so since it was included in the docs I assumed it was already supported. |
|
@mgood yeah sorry about that. We pulled the trigger on documentation too early. We updated it last week to clarify that it will be available starting 5.11. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Service discovery looks perfect for monitoring things running in kube. However, setting up another etcd cluster and keeping it up-to-date with deployments would get annoying.
Kubernetes lets you store arbitrary metadata in the
annotationson a pod, which could include the datadog check config. That would be a pretty lean and powerful setup - you could switch monitoring configurations for yourDeploymentand get a rolling deploy of it. The metadata is stored in etcd at the end, so it's essentially the same setup. The nice part is there are less moving parts - you can reuse your existing "push to prod" logic to push monitoring logic.The text was updated successfully, but these errors were encountered: