[Feature]: Add metrics and other monitoring for the signer

[djordon]

Feature - Add metrics and other monitoring for the signer

1. Description

When we're on testnet, we need alerts for all of our error level logs so that we can see our issues and fix stuff. This ticket is for setting up such alerts.

1.1 Context & Purpose

We need metrics and alerts to know what is going wrong. So let's add them.

2. Technical Details:

2.1 Acceptance Criteria:

• We have settled on services to use for logs, metrics, and alerts.
• We have alerts on error logs.
• We have error logs for fatal issues that arise.

3. Related Issues and Pull Requests (optional):

This came up in #474 (comment).

[cylewitruk]

We have settled on services to use for logs, metrics, and alerts.

Tbh if this will run in AWS I would just settle on tracing-opentelemetry and X-Ray -- the alternative cost in devops for setting up "cooler" tooling would probably outweigh its benefits.

Prometheus can now ingest OTel data, so having opentelemetry as the only exporter works great to avoid vendor lock-in as most tooling can ingest it 😄