Metrics as a first class feature

[cjellick]

We want to make it much easier for users to expose metrics. The current best practice for a user using raw k8s is to create a metrics endpoint in the application and then wire it into prometheus using CRDs.

We want to simplify that down to the user just needing to define a metrics port and path for an acorn container and acorn will take care of the wiring up.

This should also be compatible with other technologies in our stack including datadog and istio. We've seen that datadog can figure out how to scrape metrics based on prometheus annotations, but we need to figure out if that is viable in the mid-to-long term because prom itself encourages crds rather than annotation.

On the istio front, we know that istio also exposes its own metrics and "publishes" them for scraping via the prometheus annotations. In theory, if the main container (to istio's side car) also sets metrics annotations, then istio will figure out how to expose both the sidecar and main container metrics. We just need to make sure this all comes together as expected and works hollistically.

[g-linville]

Since we are still using Istio and Istio works well with the prometheus.io annotations, I am going to set this up to create those for now. We can think about Prometheus Operator custom resources in the future if that is something we also want to do.

[g-linville]

Main feature work is done. Now just one more PR for docs.

[g-linville]

Docs PR merged. Moving to Staging.

To validate this, install Acorn locally and deploy a simple app like this one:

containers: {
  "nginx": {
    image: "nginx:latest"
    ports: "80/http"
    metrics: {
      port: 80
      path: "/"
    }
    files: {
      "/usr/share/nginx/html/index.html": """
      # TYPE my_fake_metric gauge
      my_fake_metric 11111

      # TYPE my_other_fake_metric counter
      my_other_fake_metric 22222

      # TYPE my_third_metric_with_a_label gauge
      my_third_metric_with_a_label{mylabel="wow"} 33333

      """
    }
  }
}

The Pod created for this container should have the following annotations:

prometheus.io/scrape: "true"
prometheus.io/port: "80"
prometheus.io/path: "/"

[sangee2004]

Tested with v0.7.1-46-ge262f63c+e262f63c

Having the following metrics parameter defined for containers and jobs results in prometheus annotations created in the pods :

    metrics: {
      port: 80
      path: "/"
    }

Pods created for the containers and jobs have the following annotations:

prometheus.io/scrape: "true"
prometheus.io/port: "80"
prometheus.io/path: "/"