New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Gather more granular timing info #460

Open

2 tasks

bajtos opened this issue Jan 27, 2025 · 0 comments

Member

bajtos commented Jan 27, 2025

spark-evaluate was recently CPU-throttled. It may be time to re-evaluate our design choices and find a more performant design.

The first step is to get more data to identify the performance bottlenecks.

Let's improve spark-evaluate to collect timing data about all CPU-intensive steps:

fraud detection
committee evaluations
data aggregations for InfluxDB
data aggregations - Spark-specific stats
data aggregations - platform-wide stats
etc.

Tasks:

Report timings to InfluxDB
Visualise timings in Grafana

Prior art:

Time to assess measurements in round

github-project-automation bot added this to Space Meridian

bajtos mentioned this issue

M5.2 CheckerNetwork/roadmap#219

Open

60 tasks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment