[core] Collector and Checks metrics Governor

[yannmh]

Collector and Checks metrics Governor is ready for review !

What is a Limiter ?

A limiter is a rule set to monitor the metric stream.

It is defined by:

• a scope: scope where the rule applies
• a selection: selection where the rule applies
• an optional limit: limit of selections per scope.

scope and selection are defined as subset of ['name', 'agent', 'check', 'tags'] and allow to define
custom rules.
For instance
'scope' = ('name', 'check') selection = 'tags' monitors number of tags per (metric name, check)
'scope' = 'agent' selection = 'name' monitors globally the number of metrics names

Limiters keep tracks of 'active' selections (see point below), i.e. metrics's met selections on a sliding window, and count the number of metrics exceeding the limit set.

What an 'active' selection ?

A selection is considered as 'active', if it has been met at least once during the last X metrics stream batch processing, where X is a settable parameter.

What is a Governor ?

A Governor supervises multiple limiters: submitting metric stream batches for checks, and reporting when an anomaly is detected.

Governors are split in two categories

• CheckGovernor.
  Supervise check scoped limiters.
• AgentGovernor.
  Supervise wide scoped limiters.
  When a limiter hit the active selections limit, a warning is post to Datadog.

Notes

For now the Governor is ready-only and does not alterate any payload.

TO-DO

• Choose where to get Governor settings from (hardcoded for now)
• Define Governor settings
• Enable Governor report endpoint
• Add a Governor for Dogstatsd metrics

[degemer]

I think you could use the new utils/ subclass for governor.py.

[LeoCavaille]

It would probably me more flexible to have a agent_governor/report endpoint and support different "events" from the governor

[yannmh]

Good idea @degemer. Moving governor.py to utils/

[remh]

Won't be implemented.