Log warnings when too many leases are active

[armon]

Feature Request:

We should issue a warning in the log message when too many leases are active in the system. This is usually an indication that the rate in (new leases) does not match rate out (expiration / revocation) and will eventually lead to cluster degradation or failure.

Conservatively, we could use a high warning threshold like 256K leases, which is unlikely to ever be a reasonable steady state. This could potentially be made a tunable knob if necessary.

This is where the metric is currently emitted, might be good to restrict logging to once every N interval to avoid flooding.

vault/vault/expiration.go

Line 1271 in 8142b42

metrics.SetGauge([]string{"expire", "num_leases"}, float32(num))

[jefferai]

Closed by #3957