Welcome to the monitoring system of the transcode.ninja LivePeer orchestrator. This system allows us to monitor the transcoder and overall system performance closely. Our monitoring system's dashboards were based on the ones found in this forum post.
- Prometheus: Your data powerhouse for monitoring.
- Grafana: Elevate your monitoring game with dynamic visualizations.
- dcgm-exporter: Keep a keen eye on your Nvidia GPU.
- liquidctl and liquidctl-exporter: Monitor your Corsair HX1200i PSU with precision.
- node-exporter: Comprehensive system statistics at your fingertips.
- livepeer-exporter: A Prometheus exporter that fetches additional Livepeer orchestrator metrics, complementing the existing ones provided by the official Livepeer monitoring system.
Ensure your system is ready for the monitoring journey by meeting these prerequisites:
- Ubuntu 22.04 (or higher).
- Docker (optional but recommended).
- Nvidia GPU (optional).
- Nvidia driver (optional).
-
Clone this Repository with its submodules:
git clone --recurse-submodules [email protected]:transcodeninja/livepeer-setup.git
-
Launch the Monitoring Tools with Docker Compose:
docker compose up -d
-
Re-create the library panels: Unfortunately, Grafana does not yet support provisioning of library panels. You must manually re-create and re-link the library panels in the
livepeer_payements_overview.jsonandpsu_overview.jsondashboards. This is a one-time task. -
Setup alerting: The dashboards have various alerts. It's essential to configure your Grafana notification policies and contact points to ensure you receive these alerts on your preferred channel.
Important
If you're using the ufw firewall, set it to welcome incoming connections from the livepeer Docker subnet to localhost on the specific exporter ports. For instance, use this command:
sudo ufw allow in from 172.19.0.0/16 to any port 9100This opens the door for incoming traffic from the Docker network (subnet: 172.19.0.0/16) on port 9100.