Metrics/Prometheus

[yasiryagi]

I propose that we get metrics working for the indexing containers and then require all testnet indexers install and configure prometheus (we happy to work with the team on the template/docker-compose for both for server and indexing containers) and expose it to the team.
This way the team can collect the data without asking. We will be delighted to help the team setup the collection server.

[jamesbayly]

@ianhe8x ?

[Staking7pc]

In addition to server level and container level metrics it would be also good to have subquery specific metrics added to prometheus like number of queries served, current block indexed , response time etc for each indexer much like how substrate based projects expose a variety of metrics at port 9615 and 9616 by default .

[yasiryagi]

Currently we have this setup to monitor the health of host and containers
https://github.com/yasiryagi/SubQuery/tree/main/monitoring
It will be cool if we can get data from within the containers.

[mzxyz]

@yasiryagi thanks for the suggestion, we already consider to export metrics for indexer monitoring the services. Will release more news once we confirm the solution, thanks