Make CortexIngesterReachingSeriesLimit warning less sensitive

As it turns out, during normal shuffle-sharding operation, the 70% mark is often exceeded, but not by much. Therefore, this change sets the new warning mark at 75%. It also increases the `for` duration to 15m as the expected reaction time for warning alerts is usually in the order of hours, so we can as well wait a bit longer to see if the problem is transient. Signed-off-by: beorn7 <[email protected]>
grafana · Jul 27, 2021 · 0470adb · 0470adb
1 parent 0d48edf
commit 0470adb
Show file tree

Hide file tree

Showing 2 changed files with 2 additions and 1 deletion.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -29,6 +29,7 @@
 * [ENHANCEMENT] cortex-mixin: Added `alert_excluded_routes` config to exclude specific routes from alerts. #338
 * [ENHANCEMENT] Added `CortexMemcachedRequestErrors` alert. #346
 * [ENHANCEMENT] Ruler dashboard: added "Per route p99 latency" panel in the "Configuration API" row. #353
+* [ENHANCEMENT] Tweaked threshould and `for` duration for `CortexIngesterReachingSeriesLimit` warning alert. #362
 * [BUGFIX] Fixed `CortexIngesterHasNotShippedBlocks` alert false positive in case an ingester instance had ingested samples in the past, then no traffic was received for a long period and then it started receiving samples again. #308
 * [BUGFIX] Alertmanager: fixed `--alertmanager.cluster.peers` CLI flag passed to alertmanager when HA is enabled. #329
 * [BUGFIX] Fixed `CortexInconsistentRuntimeConfig` metric. #335

diff --git a/cortex-mixin/alerts/alerts.libsonnet b/cortex-mixin/alerts/alerts.libsonnet
@@ -257,7 +257,7 @@
                 (cortex_ingester_instance_limits{limit="max_series"} > 0)
             ) > 0.7
           |||,
-          'for': '5m',
+          'for': '3h',
           labels: {
             severity: 'warning',
           },