From 3caf2734c3141f50e58b760f3452b5db435d43c8 Mon Sep 17 00:00:00 2001 From: Matthias Rampke Date: Sun, 3 Mar 2024 13:32:57 +0000 Subject: [PATCH] Document common "I see no data" question This comes up frequently, because it is confusing unless you understand the intricacies of the Prometheus data and query model, CloudWatch metrics convergence, and the relevant settings. Fixes #580. Explanation adapted from https://github.com/prometheus/cloudwatch_exporter/issues/483#issuecomment-1425515511. Signed-off-by: Matthias Rampke --- README.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/README.md b/README.md index 336236f8..5ab45e40 100644 --- a/README.md +++ b/README.md @@ -145,6 +145,19 @@ For certain metrics which update relatively rarely, such as from S3, a timestamp. This is as the true timestamp from CloudWatch could be so old that Prometheus would reject the sample. +#### FAQ: I can see the metrics in `/metrics` but not in the Prometheus web console + +The metrics will be visible in Prometheus if you look more than `delay_seconds` in the past. +Try the graph view. + +This is an unfortunate result of a fundamental mismatch between CloudWatch and Prometheus. +CloudWatch metrics converge over time, that is, the value at time `T` can change up to some later time `T+dT`. +Meanwhile, Prometheus assumes that once it has scraped a sample, that is the truth, and the past does not change. + +To compensate for this, by default the exporter [delays fetching metrics](https://github.com/prometheus/cloudwatch_exporter/blob/master/README.md#timestamps), that is, it only asks for data 10 minutes later, when _almost_ all AWS services have converged. +It also reports to Prometheus that this sample is from the past. +Because Prometheus, for an instant request, only looks back 5 minutes, it never sees any data "now". + ### Special handling for certain DynamoDB metrics The DynamoDB metrics listed below break the usual CloudWatch data model.