Setup service monitoring for TSA in terraform

[haydentherapper]

Description

[jku]

It turns out we have both of these already. I'm no expert but these look like I expected:

• https://github.com/sigstore/helm-charts/tree/main/charts/tsa
• https://github.com/sigstore/scaffolding/tree/main/terraform/gcp/modules/timestamp

There is even https://github.com/sigstore/scaffolding/tree/main/cmd/tsa for setting things up in (the CI part of) scaffolding.

I think I'll close this one. We have #54 already but I will file a new issue for actually running tsa in staging: this is needed by the time we want to test rekor-tiles in staging.

[haydentherapper]

@jku Reopening to track the addition of Terraform configuration for monitoring the TSA, under https://github.com/sigstore/scaffolding/tree/main/terraform/gcp/modules/monitoring. I initially added the Terraform configuration for the TSA, but didn't add monitoring as well. I think we mostly need to copy the Rekor or Fulcio monitoring configuration, and add additional alerts based on any exported metrics from the TSA service, https://github.com/sigstore/timestamp-authority/blob/95148b257e1410c4962c32cb3b3aecffe5b2892e/pkg/api/metrics.go#L27

[jku]

I can't edit the title to reflect that but sure sounds fine to me.

So the task is actually:

setup service monitoring for TSA in terraform

These estimates based on rekor etc -- we can discuss and tweak

• metrics: fulcio/k8s_pod/restarting-failed-container and fulcio/k8s_pod/unschedulable
• define slo:
  • 0.995 goal for server-availability, all endpoints, counting 5xx errors only
  • 0.995 goal for prober-availability, all endpoints
• uptime: test an api endpoint every 60 secs
• setup alerts:
  • for k8s metrics
  • for uptime failure (five consecutive failures)

The API endpoints are:

• /api/v1/timestamp/
• /api/v1/timestamp/certchain