Agent Metrics Configuration

[BrandonS05]

Is your feature request related to a problem? Please describe.
Now that Vault Agent metrics have been released in v1.10.0 (hashicorp/vault#13675), the injector needs a way to enable and configure them. There is a strong need for those using the Vault Injector in production to have insights on Vault authentication/ connection issues from the injected agent.

Describe the solution you'd like
A means to configure metrics on Vault Agent via Injector annotations and environment variables

Describe alternatives you've considered
The "vault.hashicorp.com/agent-configmap" annotation allows for mounting a custom Agent config in place of relying on annotations. Redesigning our existing pipeline to generate a unique configmap for each service's agent, just to enable metrics configuration, is less than ideal though.

[tvoran]

Thanks for opening this! It's definitely something we'd like have supported, as you can imagine.

Since we don't yet have a way to expose a metrics-only endpoint on agent, we'll want to make sure the listener is only on localhost (as you mentioned in #329). And since there are a lot of options for the telemetry stanza, it probably needs to be controlled by a more flexible config annotation style, similar to the auth-config annotation if possible.

[Freyert]

This is a very high value add for me so I want to see how feasible a contribution here is. Given the previously abandoned PR I want to make sure I get the requirements right.

1. Without the ability to open a separate port for telemetry the injector should only listen on localhost. One would also need to inject some sort of proxy into the pod in this case. Therefore I want to know is there a ticket to allow Vault Agent to open telemetry on a separate port?
2. Since telemetry is its own complex stanza, follow the design for auth-config annotation.

Using the Prometheus also seems to come with a lot of caveats around auth https://www.vaultproject.io/docs/configuration/telemetry#prometheus

---

I was looking into this myself. Before, diving too deeply into it I was wondering if vault.hashicorp.com/agent-configmap replaces all the injectors configuration? Would I need to redefine all stanza? Or could I just add the telemetry stanza?

It looks like I would indeed need to redefine all the other stanzas including the auth stanzas. 😨

vault-k8s/agent-inject/agent/container_sidecar.go

Lines 62 to 69 in bd9da8f

	if a.ConfigMapName != "" {
	volumeMounts = append(volumeMounts, corev1.VolumeMount{
	Name: configVolumeName,
	MountPath: configVolumePath,
	ReadOnly: true,
	})
	arg = fmt.Sprintf("touch %s && vault agent -config=%s/config.hcl", TokenFile, configVolumePath)
	}

Normally the Agent reads its config from an environment variable built by all the annotations. So annotations for a telemetry stanza would be key.

vault-k8s/agent-inject/agent/container_sidecar.go

Line 18 in bd9da8f

DefaultContainerArg = "echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json"

[Freyert]

Since we don't yet have a way to expose a metrics-only endpoint on agent, we'll want to make sure the listener is only on localhost (as you mentioned in #329).

^ @tvoran given your concern here, would it be OK to just contribute a way to configure a telemetry stanza that, also hard codes its listener to localhost? As long as it works similarly to the auth annotations?

I think that would at least give folks the option to expose the metrics via some sort of proxy internal to the pod. I think the telemetry/listener stanzas need some considerable rethinking to the core product to allow isolated telemetry "listeners".

---

Some notes on the auth annotations:

1. It can support a single generic auth stanza
   1. ✅ there are many telemetry backends to cater to prometheus, stackdriver, datadog, etc.
2. The injector only supports a single auth stanza
   1. ⚠️ there can be multiple telemetry stanzas (from what I know ❓)
   2. Will probably just need to "id" each telemetry stanza.

[VioletHynes]

Hey! I can't speak to the changes required to the K8S/Helm side of things, but we do have a metrics only option for listeners: https://developer.hashicorp.com/vault/docs/agent#listener-stanza

In particular, this would solve the "ability to open a separate port for telemetry" issue, as there now is a way to expose a metrics only listener for Vault Agent.

[LS80]

I think this was fixed by #413?

But we still can't scrape the metrics from Prometheus because the agent only listens on localhost. So we would need another annotation (maybe vault.hashicorp.com/agent-metrics-listener-port) to configure a metrics listener stanza.

listener "tcp" {
  address = "0.0.0.0:8080"
  role = "metrics_only"
}

A workaround would be a simple nginx sidecar proxy, but that's not always possible if the pod is created by a controller that doesn't allow configuring additional containers.

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  nginx.conf: |
    events {}
    http {
        server {
            listen 8201;

            location /metrics {
                proxy_pass http://127.0.0.1:8200/agent/v1/metrics?format=prometheus;
            }
        }
    }

[LS80]

Using socat is probably the simplest workaround.

- name: vault-agent-metrics-proxy
  image: alpine/socat:1.8.0.0
  command:
  - socat
  - TCP4-LISTEN:8201
  - TCP4:127.0.0.1:8200
  ports:
  - containerPort: 8201
    name: vault-metrics

[LS80]

@tvoran What do you think of something like #698? I will wait for feedback before going any further with it.