Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dashboard: add two new endpoints to dashboard for Nomad examples #6

Merged
merged 1 commit into from
Apr 21, 2020
Merged

dashboard: add two new endpoints to dashboard for Nomad examples #6

merged 1 commit into from
Apr 21, 2020

Conversation

shoenig
Copy link
Member

@shoenig shoenig commented Mar 23, 2020

The Nomad team is rolling out configuration of Envoy expose paths
for its Consul Connect integration. In Nomad's context, configuring
expose paths allows for "punching a hole" through the network namespace
that Nomad automatically creates for Connect enabled tasks. To demonstrate
this capability, this CR updates the dashboard-service to expose 2
new endpoints. Nomad makes extensive use of these two services throughout
its documentation as a typical example of a Connect setup.

  • /health/api: provides a real health-check against the connectivity
    with the back-end API service. Consecutive connection failures
    are counted, and if non-zero, the endpoint returns a 503. If
    the most recent connection attempt was success, returns 200.

  • /metrics: simply exposes Go's built-in expvar Handler, which
    produces a JSON response full of Go runtime memory metrics and
    other such data.

The existing endpoints are left as-is.

Example usage:

launch counting service
$ PORT=8888 go run main.go

launch dashboard service

$ PORT=9999 COUNTING_SERVICE_URL='http://localhost:8888' go run main.go
show /metrics endpoint
$ curl  -s localhost:9999/metrics | jq . | head -n 10
{
  "cmdline": [
    "/tmp/go-build318152207/b001/exe/main"
  ],
  "memstats": {
    "Alloc": 1193440,
    "TotalAlloc": 1193440,
    "Sys": 71387144,
    "Lookups": 0,
    "Mallocs": 2822,
...
show /health/api endpoint with counting service up & running
$ curl -w '\ncode: %{response_code}\n' -s localhost:9999/health/api
ok
code: 200
show /health/api endpoint with counting service shutdown
$ curl -w '\ncode: %{response_code}\n' -s localhost:9999/health/api
failures: 3
code: 503

@shoenig
dashboard: add two new endpoints to dashboard for Nomad examples
4aa0d32 
The Nomad team is rolling out configurability of Envoy expose paths
for its Consul Connect integration. In Nomad's context, configuring
expose paths allows for "punching a hole" through the network namespace
that Nomad automatically creates for Connect enabled tasks. To demonstrate
this capability, this CR updates the dashboard-service to expose 2
new endpoints.

- `/health/api`: provides a real healthcheck against the connectivity
	with the backend API service. Consecutive connection failures
	are counted, and if non-zero, the endpoint returns a 503. If
	the most recent connection attempt was success, returns 200.

- `/metrics`: simply exposes Go's built-in `expvar` Handler, which
	produces a JSON response full of Go runtime memory metrics and
	other such data.

The existing endpoints are left as-is.

Example usage:

```bash

$ PORT=8888 go run main.go
```

```bash

$ PORT=9999 COUNTING_SERVICE_URL='http://localhost:8888' go run main.go
```

```bash

$ curl  -s localhost:9999/metrics | jq . | head -n 10
{
  "cmdline": [
    "/tmp/go-build318152207/b001/exe/main"
  ],
  "memstats": {
    "Alloc": 1193440,
    "TotalAlloc": 1193440,
    "Sys": 71387144,
    "Lookups": 0,
    "Mallocs": 2822,
...
```

```bash

$ curl -w '\ncode: %{response_code}\n' -s localhost:9999/health/api
ok
code: 200
```

```bash

$ curl -w '\ncode: %{response_code}\n' -s localhost:9999/health/api
failures: 3
code: 503
```
@shoenig shoenig requested review from topfunky and angrycub March 23, 2020 20:02
shoenig added a commit to hashicorp/nomad that referenced this pull request Mar 23, 2020
@shoenig
docs: add documentation for proxy.expose configuration
9233ee6 
This changeset adds documentation changes for `proxy.expose` and
the `proxy.expose.path` stanzas in the jobspec.

The examples are centered around proposed changes for the "countdash"
`dashboard-service` in [demo-consul-101](github.com/hashicorp/demo-consul-101/pull/6).
The dashboard service will now serve two additonal endpoints

- `/health/api`
- `/metrics`

which should serve nicely as expose-able paths.
shoenig added a commit to hashicorp/nomad that referenced this pull request Mar 26, 2020
@shoenig
connect: enable automatic expose paths for individual group service c…
2b43a44 
…hecks

Part of #6120

Building on the support for enabling connect proxy paths in #7323, this change
adds the ability to configure the 'service.check.expose' flag on group-level
service check definitions for services that are connect-enabled. This is a slight
deviation from the "magic" that Consul provides. With Consul, the 'expose' flag
exists on the connect.proxy stanza, which will then auto-generate expose paths
for every HTTP and gRPC service check associated with that connect-enabled
service.

A first attempt at providing similar magic for Nomad's Consul Connect integration
followed that pattern exactly, as seen in #7396. However, on reviewing the PR
we realized having the `expose` flag on the proxy stanza inseperably ties together
the automatic path generation with every HTTP/gRPC defined on the service. This
makes sense in Consul's context, because a service definition is reasonably
associated with a single "task". With Nomad's group level service definitions
however, there is a reasonable expectation that a service definition is more
abstractly representative of multiple services within the task group. In this
case, one would want to define checks of that service which concretely make HTTP
or gRPC requests to different underlying tasks. Such a model is not possible
with the course `proxy.expose` flag.

Instead, we now have the flag made available within the check definitions themselves.
By making the expose feature resolute to each check, it is possible to have
some HTTP/gRPC checks which make use of the envoy exposed paths, as well as
some HTTP/gRPC checks which make use of some orthongonal port-mapping to do
checks on some other task (or even some other bound port of the same task)
within the task group.

Given this example,

group "server-group" {
  network {
    mode = "bridge"
    port "forchecks" {
      to = -1
    }
  }

  service {
    name = "myserver"
    port = 2000

    connect {
      sidecar_service {
      }
    }

    check {
      name     = "mycheck-myserver"
      type     = "http"
      port     = "forchecks"
      interval = "3s"
      timeout  = "2s"
      method   = "GET"
      path     = "/classic/responder/health"
      expose   = true
    }
  }
}

Nomad will automatically inject (via job endpoint mutator) the
extrapolated expose path configuration, i.e.

expose {
  path {
    path            = "/classic/responder/health"
    protocol        = "http"
    local_path_port = 2000
    listener_port   = "forchecks"
  }
}

Documentation is coming in #7440 (needs updating, doing next)

Modifications to the `countdash` examples in hashicorp/demo-consul-101#6
which will make the examples in the documentation actually runnable.

Will add some e2e tests based on the above when it becomes available.
@shoenig shoenig mentioned this pull request Mar 26, 2020
Merged
shoenig added a commit to hashicorp/nomad that referenced this pull request Mar 27, 2020
@shoenig
docs: add documentation for proxy.expose configuration
dcbfa67 
This changeset adds documentation changes for the new
`proxy.expose` stanza as well as the `check.expose` parameter.

The examples are centered around proposed changes for the "countdash"
`dashboard-service` in [demo-consul-101](github.com/hashicorp/demo-consul-101/pull/6).
The dashboard service will now serve two additonal endpoints

- `/health/api`
- `/metrics`

which should serve nicely as expose-able paths.
shoenig added a commit to hashicorp/nomad that referenced this pull request Mar 30, 2020
@shoenig
connect: enable automatic expose paths for individual group service c…
3223686 
…hecks

Part of #6120

Building on the support for enabling connect proxy paths in #7323, this change
adds the ability to configure the 'service.check.expose' flag on group-level
service check definitions for services that are connect-enabled. This is a slight
deviation from the "magic" that Consul provides. With Consul, the 'expose' flag
exists on the connect.proxy stanza, which will then auto-generate expose paths
for every HTTP and gRPC service check associated with that connect-enabled
service.

A first attempt at providing similar magic for Nomad's Consul Connect integration
followed that pattern exactly, as seen in #7396. However, on reviewing the PR
we realized having the `expose` flag on the proxy stanza inseperably ties together
the automatic path generation with every HTTP/gRPC defined on the service. This
makes sense in Consul's context, because a service definition is reasonably
associated with a single "task". With Nomad's group level service definitions
however, there is a reasonable expectation that a service definition is more
abstractly representative of multiple services within the task group. In this
case, one would want to define checks of that service which concretely make HTTP
or gRPC requests to different underlying tasks. Such a model is not possible
with the course `proxy.expose` flag.

Instead, we now have the flag made available within the check definitions themselves.
By making the expose feature resolute to each check, it is possible to have
some HTTP/gRPC checks which make use of the envoy exposed paths, as well as
some HTTP/gRPC checks which make use of some orthongonal port-mapping to do
checks on some other task (or even some other bound port of the same task)
within the task group.

Given this example,

group "server-group" {
  network {
    mode = "bridge"
    port "forchecks" {
      to = -1
    }
  }

  service {
    name = "myserver"
    port = 2000

    connect {
      sidecar_service {
      }
    }

    check {
      name     = "mycheck-myserver"
      type     = "http"
      port     = "forchecks"
      interval = "3s"
      timeout  = "2s"
      method   = "GET"
      path     = "/classic/responder/health"
      expose   = true
    }
  }
}

Nomad will automatically inject (via job endpoint mutator) the
extrapolated expose path configuration, i.e.

expose {
  path {
    path            = "/classic/responder/health"
    protocol        = "http"
    local_path_port = 2000
    listener_port   = "forchecks"
  }
}

Documentation is coming in #7440 (needs updating, doing next)

Modifications to the `countdash` examples in hashicorp/demo-consul-101#6
which will make the examples in the documentation actually runnable.

Will add some e2e tests based on the above when it becomes available.
shoenig added a commit to hashicorp/nomad that referenced this pull request Mar 30, 2020
@shoenig
docs: add documentation for proxy.expose configuration
a1a49d4 
This changeset adds documentation changes for the new
`proxy.expose` stanza as well as the `check.expose` parameter.

The examples are centered around proposed changes for the "countdash"
`dashboard-service` in [demo-consul-101](github.com/hashicorp/demo-consul-101/pull/6).
The dashboard service will now serve two additonal endpoints

- `/health/api`
- `/metrics`

which should serve nicely as expose-able paths.
shoenig added a commit to hashicorp/nomad that referenced this pull request Mar 31, 2020
@shoenig
connect: enable automatic expose paths for individual group service c…
e63f13a 
…hecks

Part of #6120

Building on the support for enabling connect proxy paths in #7323, this change
adds the ability to configure the 'service.check.expose' flag on group-level
service check definitions for services that are connect-enabled. This is a slight
deviation from the "magic" that Consul provides. With Consul, the 'expose' flag
exists on the connect.proxy stanza, which will then auto-generate expose paths
for every HTTP and gRPC service check associated with that connect-enabled
service.

A first attempt at providing similar magic for Nomad's Consul Connect integration
followed that pattern exactly, as seen in #7396. However, on reviewing the PR
we realized having the `expose` flag on the proxy stanza inseperably ties together
the automatic path generation with every HTTP/gRPC defined on the service. This
makes sense in Consul's context, because a service definition is reasonably
associated with a single "task". With Nomad's group level service definitions
however, there is a reasonable expectation that a service definition is more
abstractly representative of multiple services within the task group. In this
case, one would want to define checks of that service which concretely make HTTP
or gRPC requests to different underlying tasks. Such a model is not possible
with the course `proxy.expose` flag.

Instead, we now have the flag made available within the check definitions themselves.
By making the expose feature resolute to each check, it is possible to have
some HTTP/gRPC checks which make use of the envoy exposed paths, as well as
some HTTP/gRPC checks which make use of some orthongonal port-mapping to do
checks on some other task (or even some other bound port of the same task)
within the task group.

Given this example,

group "server-group" {
  network {
    mode = "bridge"
    port "forchecks" {
      to = -1
    }
  }

  service {
    name = "myserver"
    port = 2000

    connect {
      sidecar_service {
      }
    }

    check {
      name     = "mycheck-myserver"
      type     = "http"
      port     = "forchecks"
      interval = "3s"
      timeout  = "2s"
      method   = "GET"
      path     = "/classic/responder/health"
      expose   = true
    }
  }
}

Nomad will automatically inject (via job endpoint mutator) the
extrapolated expose path configuration, i.e.

expose {
  path {
    path            = "/classic/responder/health"
    protocol        = "http"
    local_path_port = 2000
    listener_port   = "forchecks"
  }
}

Documentation is coming in #7440 (needs updating, doing next)

Modifications to the `countdash` examples in hashicorp/demo-consul-101#6
which will make the examples in the documentation actually runnable.

Will add some e2e tests based on the above when it becomes available.
shoenig added a commit to hashicorp/nomad that referenced this pull request Apr 1, 2020
@shoenig
docs: add documentation for proxy.expose configuration
44f884e 
This changeset adds documentation changes for the new
`proxy.expose` stanza as well as the `check.expose` parameter.

The examples are centered around proposed changes for the "countdash"
`dashboard-service` in [demo-consul-101](github.com/hashicorp/demo-consul-101/pull/6).
The dashboard service will now serve two additonal endpoints

- `/health/api`
- `/metrics`

which should serve nicely as expose-able paths.
@angrycub angrycub merged commit 68ee803 into master Apr 21, 2020
@angrycub angrycub deleted the f-more-dashboard-endpoints branch April 21, 2020 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@angrycub angrycub angrycub approved these changes

@topfunky topfunky Awaiting requested review from topfunky

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@shoenig @angrycub