Add health checks to all controllers

[andrewrynhard]

We should add readiness and liveness problem to all controllers.

[abckey]

any update on this, we met this problem again:

E0915 13:05:47.387749       1 leaderelection.go:357] Failed to update lock: Put "https://10.255.0.1:443/api/v1/namespaces/sidero-system/configmaps/controller-leader-election-sidero-controller-manager": context
deadline exceeded
I0915 13:05:47.387945       1 leaderelection.go:278] failed to renew lease sidero-system/controller-leader-election-sidero-controller-manager: timed out waiting for the condition
2021-09-15T13:05:47.388Z        DEBUG   controller-runtime.manager.events       Normal  {"object": {"kind":"ConfigMap","apiVersion":"v1"}, "reason": "LeaderElection", "message": "sidero-controller-manager-76fc9
d4ddf-fx9vk_ec017c02-6eee-443f-b100-81d5560091e6 stopped leading"}
2021-09-15T13:05:47.388Z        ERROR   setup   problem running manager {"error": "leader election lost"}
golang.org/x/sync/errgroup.(*Group).Go.func1
        /.cache/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57
2021-09-15T13:05:47.389Z        INFO    controller      Stopping workers        {"reconcilerGroup": "metal.sidero.dev", "reconcilerKind": "Server", "controller": "server"}
2021-09-15T13:05:47.389Z        INFO    controller      Stopping workers        {"reconcilerGroup": "metal.sidero.dev", "reconcilerKind": "Environment", "controller": "environment"}
2021-09-15T13:05:47.389Z        INFO    controller      Stopping workers        {"reconcilerGroup": "metal.sidero.dev", "reconcilerKind": "ServerClass", "controller": "serverclass"}

[smira]

We are going to take a look today. It's a bug that controller doesn't crash

[smira]

I don't think liveness check is the right answer here though, as controller being working is hard to check from outside.