Healthcheck Plugin

[subnetmarco]

Provide an healthcheck plugin that can do tcp or http healthchecks on one or more API.

Here is a few problems:

• Healthchecks need to run on one machine per cluster, otherwise a 200-nodes Kong cluster will DDOS the final API.
• This means that somehow Kong needs to be aware of the cluster, and elect a master healthcheck node that will run the healthchecks.
• If this node for some reason gets shut down, the cluster needs to be aware of that and elect a new master.
• This also means that changing the healthcheck configuration on one node, needs to be communicated to the master node that is unaware of the change.

[subnetmarco]

The same problem will happen with a billing plugin: only one node needs to execute the billing functionality.

[thibaultcha]

What about pinging from a random node every time?

[subnetmarco]

Since each node is not aware of each other, there is no coordination mechanism between nodes. This is because we decided to keep cluster awareness off from this early releases, but to have a better system we would need to implement some sort of cluster awareness at one point. Usually this can be done with Apache Zookeeper, or Serfdom, or consul.io. Once a plugin is being installed and an async timer is being started to do the health checks, it will run by default on every node.

The first implementation of the health check can run on every node, but we need to address cluster awareness sooner than later to fix this and other problems.

For example, using features like https://serfdom.io/intro/getting-started/user-events.html will also help us implementing proper invalidations #15 (as opposed to having intermediate Cassandra tables for coordination).

[hutchic]

Personally I'd close this and let the service discovery tool of the user's choosing do the health checks unless there's a use case for health checks that isn't closely related to service discovery I'm unaware of

https://www.consul.io/docs/agent/http/health.html
https://github.com/airbnb/nerve
https://github.com/Netflix/eureka/
http://www.cloudera.com/documentation/enterprise/5-5-x/topics/cm_ht_zookeeper.html

[Tieske]

I think all of this is too complex. Architecturally we need to treat each node as a single system whenever possible. No intra-node communications if we can prevent it.

With the implementation of the balancer_by_lua directives we already have health info available based on the request results. See this code

This uses the get_last_failure method.

This obviously uses 'passive' monitoring (based on failed actual requests) as opposed to the initial post above suggesting 'active' monitoring by firing explicit probe-requests.

[shaowin16]

Where is the plugin's url?
Can you attach it?

[Tieske]

@shaowin16 it hasn't been added yet. Probably for 0.11

[r-alekseev]

Healthchecks need to run on one machine per cluster, otherwise a 200-nodes Kong cluster will DDOS the final API.
This means that somehow Kong needs to be aware of the cluster, and elect a master healthcheck node that will run the healthchecks.

I think all of this is too complex. Architecturally we need to treat each node as a single system whenever possible. No intra-node communications if we can prevent it.

To avoid DDOS and intra-node communications, we can invent a setting that represents a chance of a node to send a healthcheck at the end of a time interval.

f.e.:
interval = 1 second
chance = 1/200

Each node every second generate a random number from 1 to 200.
Some bingo node(s) will send a healthcheck.

[Tieske]

Even in that case, if the healthcheck results in a state change, we're back to cross-cluster communications.

As an update, healthchecks is actively being worked on at the moment.

[coopr]

Health checks are scheduled to be released in Kong Community Edition version 0.12

[Tieske]

since 0.12 is released, closing this.