SRE-96: Add pipeline to send application metrics to Honeycomb #12
Conversation
| - job_name: {{ .Release.Namespace }} | ||
| scrape_interval: 15s | ||
| static_configs: | ||
| - targets: [""] # all the desired pods or whatever |
There was a problem hiding this comment.
I'm hoping there's some template variable that Argo will fill in for me to give me the list of all the pods in the namespace, so we can point directly to all of them. (Side note: will scraping all that overwhelm the collector?)
If not, we can add a variable to the values.yaml file in each app's Helm chart listing the pods (nodes?) we want to scrape, and then have Argo populate the config from that.
There's gonna be some variable interpolation no matter what to get the specific urls, but I'm hoping we can do it with minimal changes required in the app charts.
There was a problem hiding this comment.
So this might not be possible with our Prometheus setup until this PR gets merged: open-telemetry/opentelemetry-collector-contrib#6344
| endpoint: "api.honeycomb.io:443" | ||
| headers: | ||
| "x-honeycomb-team": "${HONEYCOMB_TEAM}" | ||
| "x-honeycomb-dataset": "${HONEYCOMB_DATASET}" |
There was a problem hiding this comment.
| "x-honeycomb-dataset": "${HONEYCOMB_DATASET}" | |
| "x-honeycomb-dataset": "app-metrics" |
We probably want to send prom data to a different dataset at first just to see what it looks like. If we do this, I'll have to create the dataset manually first, since the Honeycomb keys being used won't have dataset creation permissions.
No description provided.