Allow specifying a check ID within services to link checks with services

[cruatta]

In regards to #740, I've been thinking about a more elegant way to link services and checks. Since the health checks defined in services are not complete health checks, why not allow linking between checks and services by specifying the ID of a check in a service definition?

Example:

{
  "check": {
    "id": "mem-util",
    "name": "Memory utilization",
    "script": "/usr/local/bin/check_mem.py",
    "interval": "10s"
  }
}

{
  "service": {
    "name": "redis",
    "tags": ["master"],
    "address": "127.0.0.1",
    "port": 8000,
    "checks": [
      {
        "id": "mem-util"
      }
    ]
  }
}

I think this adds a number of benefits:

1. It encourages reuse of health checks and reduces the size of service definitions
2. Checks in services are full health checks with more meaningful IDs and names which helps during debugging by an operator
3. It reduces the number of checks that have to be run. A service can simply lookup the latest state of a check to get its own state.

[ryanuber]

Hey @cruatta,

For checks which apply to the system as a whole (like memory or cpu utilization, etc), these should actually just not be associated with any service. If they are associated with the node only, when they start failing, all services on that node are considered "unhealthy" and are not returned from the service catalog when using a passing filter, and will also not be returned from the DNS interface. Using node-scoped checks for general system-level tests should allow deduplicating most things.

Does that help?

[cruatta]

Hey Ryan. Thanks for the response. I can see how you're encouraging reuse through node level checks. A memory check was just an arbitrary example and probably not the best since it is definitely something you'd want at the node level. It would still be a big improvement to consul though if there was a way to specify an ID and Name for checks defined in the service (as mentioned in #740). That aside, I'll close out this particular issue as it was just to hear your thoughts on this approach.