Unable to put health check in task grouped with Consul Connect service.

[BenLuchterhand]

I'm trying to load balance a simple API through Fabio. I had accomplished it simply by placing the right health check in the API's task stanza, but since I switched to using Consul Connect (and using a bridged network mode) I've had to move it all outside of the task causing both the Proxy and the API getting routed through Fabio.

Ideally I'd like to move the tag and implement a health check within the API task but I can't seem to do it without encountering the error I posted in the "Issue" section. This would prevent the proxy from also getting routed. Is there any way to do this or am I trying to do something impossible? I'm very new to this sort of development, I really appreciate your help and patience!

EDIT: The group I'm referring to is the "api" group about half way down the Job file.

Nomad version

Nomad v0.10.4 (f750636)

Issue

Placing the service and health check within the API task itself causes a "failed to create container: API error (400): conflicting options: port publishing and the container type network mode" error, so to route through Fabio I placed them where they are currently in the api group outside of the task.

Job file (if appropriate)

job "apidb" {
	datacenters = ["dc1"]

	constraint {
		attribute = "${attr.kernel.name}"
		value = "linux"
	}

	update {
		stagger = "10s"

		max_parallel = 1
	}

  group "mongo" {
    # Specify the number of these tasks we want.
    count = 1
		network {
		  mode = "bridge"
		}

		service {
			name = "mongo-acc"
			port = "27017"							# Can be any port? Doesn't need to match mongo?
			connect {
				sidecar_service {
					proxy {
						local_service_address = "${attr.unique.network.ip-address}"
					}
				}
			}
		}

		task "mongo" {
			driver = "docker"

			config {
				image = "mongo:4.0.4"
			}

			# Specify the maximum resources required to run the task,
			# include CPU, memory, and bandwidth.
			resources {
				cpu    = 500 # MHz
				memory = 128 # MB

				network {
					mbits = 100
				}
			}
		}
	}

  group "api" {
		network {
			mode = "bridge"
			port "http" {
				to = 5000
			}
		}

		service {
			name = "api-frontend"
			port = "http"
			tags = ["urlprefix-/"]
			check {
        type     = "http"
        path     = "/"
        interval = "30s"
        timeout  = "5s"
      }

			connect {
				sidecar_service{
					proxy {
						upstreams {
							destination_name = "mongo-acc"
							local_bind_port = 27017
						}
					}
				}
			}
		}

		task "api" {
			driver = "docker"
			env {
				MONGO_IP = "${NOMAD_UPSTREAM_ADDR_mongo_acc}"
			}

			config {
        image = "************************"
				port_map {
            http = 5000 # points the port created by nomad to the port listening by service (5000)
        }
      }

			resources {
				cpu = 500 # 500 MHz
				memory = 128 # 128MB
				network {
					mbits = 1
				}
			}

			logs {
			    max_files = 10
			    max_file_size = 15
			}

			kill_timeout = "10s"
		}
	}
}

[shoenig]

Hey @BenLuchterhand thanks for taking the time to report this! The ability to define service checks (and more generally, exposed paths) in Connect-enabled group-level services should be coming in Nomad v0.11 (soon!). The parent ticket tracking the effort is #6120.

[BenLuchterhand]

Awesome, thank you very much @shoenig!

[BenLuchterhand]

@shoenig I am playing around with the new Nomad 0.11 beta and I think I got it working based on other threads attached to the ticket you provided. Is there any way to append tags with this system to make it discoverable by Fabio?

expose {
                               path {
			          path            = "/"
			          protocol        = "http"
			          local_path_port = 5000
			          listener_port   = "http"
			        }
		}

[shoenig]

Hey @BenLuchterhand I just updated the docs with examples for exposing task-group service checks (no need to have the underlying expose.path definition unless you're doing something other than exposing checks). They aren't published yet, but a preview is available.

Gathering what you have above, I suspect something like this should work

job "example" {
  datacenters = ["dc1"]

  group "api" {
    network {
      mode = "bridge"
      port "healthcheck" {
        to = -1 # let the "exposed" port be dynamic, reference $NOMAD_PORT_healthcheck
      }
    }

    service {
      name = "api-frontend"
      port = "5000" # useful only inside the network namespace, is what the task listens to
      tags = ["urlprefix-/"]

      connect {
        sidecar_service {
          proxy {
            upstreams {
              destination_name = "mongo-acc"
              local_bind_port  = 27017 # unrelated, mongo stuff
            }
          }
        }
      }

      check {
        type     = "http"
        name     = "api-frontend-health"
        port     = "healthcheck" # the exposed port, used by Consul outside the network namespace
        path     = "/"           # ideally only want to expose a minimal path, (e.g. /health)
        expose   = true          # new!
        interval = "30s"
        timeout  = "5s"
      }
    }

    /* ... task ... */
  }
}

Is there any way to append tags with this system to make it discoverable by Fabio?

I'm not quite sure what you mean, tags are a part of the service definition, and the checks are already inherently tied to those.

[BenLuchterhand]

That is perfect! Thank you so much!

[spuder]

I am also having difficulty trying to get heath checks working on consul connected jobs on nomad 0.11.0.

For anyone else struggling, I documented the one job that I did get working here https://discuss.hashicorp.com/t/consul-connect-with-health-checks/7591

I suspect there is a bug in the http check because I have several other jobs that should work, but as soon as I add the health check; consul connect stops working (despite the heath check being green). If I can get a reliably reproducible job I will open a new issue and link it here.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.