Monitoring epic, Oct 2017 … Sep 2018

[darkk]

This ticket tracks enhancements to OONI infra monitoring through Oct 2017 … Sep 2018.

• replace basic authentication in Prometheus with client SSL cert validation -- 23883a2 & Ceph OSD crashes cause IO stall at HKG #155
• configure cross-monitoring with another Prometheus -- 3bf09a7
• check that all inventory hosts are monitored -- d226bd1
• improve disk space alerts for filesystems over 50GiB -- 7c57b5b
• monitor rare periodic jobs with push-gateway (e.g. hellais-nas backup) -- d6ba8fc
• monitor IO stalls at GH, iowait% -- 085fd5a
• monitor IO stalls at GH, seeksample -- e342b1d
• define "canonical" TSV log format for nginx -- edae501
• monitor node_exporter/textfile liveness -- 7eff51c
• monitor nginx upstream request latency -- 6e65c46
• monitor airflow task failures -- 3f7cfac
• speedup DiskWillFillInAWeek rule -- fd63181
• check that all GH host are stored in inventory -- 8a12b5a
• improve DiskWillFillInAWeek alerts -- 01daac8, b1b3995
• monitor if OONI Website mirrors diverge -- f4841ef
• deliver email alerts to MX (without smarthost) -- 2d220b2 DNS issues at HKG cause pipieline failure #159
• monitor CNAME shadowing other DNS resource records -- 2479988
• monitor Slack-IRC gateway liveness -- b30e0d2
• improve alerts on non-critical events -- 01bf91a, Insistent notifications for urgent actionable alerts #158, Separate critical alerts from non-critical alerts #182
• improve links in airflow alerts Fix links in alert emails #153, Cleanup prometheus deployment #132
• improve links in AlertManager alerts b.web-connectivity.th down for 7.6 hours #128, Fix links in alert emails #153, Cleanup prometheus deployment #132
• ungroup AlertManager mails c.web-connectivity.th down for 17.5 hours #211
• monitor number of systemd services in failed state Not all push notifications are sent to android and iOS devices #220
• investigate dockerd monitoring Not all push notifications are sent to android and iOS devices #220
• investigate running docker containers as systemd services Not all push notifications are sent to android and iOS devices #220
• monitor anomalous and high CPU & NET usage, severity=info debuilder.infra.ooni.io compromised month ago #135 Ceph OSD crashes cause IO stall at HKG #155
• monitor memory pressure c.web-connectivity.th down for 17.5 hours #211
• monitor memory usage #101
• monitor nodes with ICMP ping to distinguish stuck userspace from bad network Ceph OSD crashes cause IO stall at HKG #155
• verify proper export of Disk IO stats Ceph OSD crashes cause IO stall at HKG #155
• steal AlertManager health checks from gitlab explorer down for ~ hour, prometheus down for ~ week #189

[darkk]

dockerd monitoring has at least one metric that sounds interesting: engine_daemon_container_actions_seconds_count{action="start"} that becomes rate of container restarts. Also, docker seems to implement backoff while restarting, so the restart loop is not "naive" busy loop.
But prometheus exporter should be explicitly enabled.

[darkk]

monitor number of systemd service restarts-per-minute

That's impossible with node_exporter 0.16.0 as soon as there are no metrics exported that can express that event. Seems, unreleased version of the node_exporter has alike features, but it's not in "stable" release yet. But frequent restarters are not "active" services so it's probably okay.

monitor number of systemd services in failed state

node_systemd_units in node_exporter=0.16.0 shows that and also exports meta-self-test is-system-running as node_systemd_system_running.

Exporting systemd metrics should be done carefully as most of units are useless (*.device, *.mount and so on).

[darkk]

investigate running docker containers as systemd services #220

Seems, it may be useful for graceful shutdown #172

[hellais]

monitor number of systemd service restarts-per-minute

I also commented in #220, but posting it here too:

https://www.robustperception.io/alerting-on-crash-loops-with-prometheus

This does require that the to-be-monitored process uses some prometheus aware library. This is the case for gorush.

[SuperQ]

I'll be releasing a new node_exporter with the systemd restarts metric in the next week or so.

[darkk]

@hellais I would love to see it done in a more generic way as gorush is just a single service out of dozen. That's why I aimed for systemd & docker metrics.

[SuperQ]

@darkk IMO, directly instrumenting is extremely valuable. Even just adding the default Prometheus client_golang for example gives you a number of process metrics, including Go internals, process CPU, RSS, etc.

It also gives you the standard Prometheus up that can be used to discover down tasks.

[darkk]

node_systemd_system_running

For some reason many hosts had no dbus package that is needed to access systemd from an unprivileged user (while all other systemd-enabled hosts had it!). I'm recording the list here for historical purposes:

• amsapi.ooni.nu
• amsdataproxy.ooni.nu
• amsmetadb.ooni.nu
• analytics.ooni.io
• get.ooni.io
• hkgcollectora.ooni.nu
• hkgjump.ooni.nu
• hkgmetadb.infra.ooni.io
• hkgwebconnectivitya.ooni.nu
• jupyter.ooni.io
• labs.ooni.io
• msg.ooni.io
• run.ooni.io
• shwdc.ooni.io
• ssdams.infra.ooni.io
• wiki.ooni.io