benchmark machine

Development plan:

1. MVP:

   Queue and runs exist on the benchmark server, which handles everything

2. Dedicated queue

   To avoid a web server running in the background, have a VM or so that maintains the queue

3. Cancellation

   Improve UX by allowing cancellation and other niceties

Usage

Eventually, we’ll just have a project-wide webhook like this. For now, if you want to test:

1. Add a asv config to your project (either the project root or a benchmarks directory)
2. Add a webhook to your scverse project with these webhook settings, i.e.
   • Content type: application/json
   • Let me select individual events → Pull Requests
3. Add a label benchmark to a PR authored by a trusted user.
4. Watch scverse-benchmarks add and update a comment with the PR’s performance impact.

MVP Setup

All these currently assume you have a <user> login with sudo rights on the scvbench server.

Debugging

• Use journalctl -u benchmark -f on the server to tail the logs of the service.
• Check GitHub’s page for Hook deliveries.

One-time server setup

1. As the benchmarker user, install micromamba, then:

   micromamba create -n asv -c conda-forge conda mamba virtualenv asv
   micromamba run -n asv asv machine --yes

   (use micromamba activate asv to make asv available in your PATH)

2. Update LoadCredentialEncrypted lines in benchmark.service using

   sudo systemd-creds encrypt --name=webhook_secret secret.txt -
   sudo systemd-creds encrypt --name=app_key app-key.pem -
   shred secret.txt app-key.pem

3. Copy the benchmark.service file to the system, enable and start the service:

   $ rsync benchmark.service <user>@scvbench:
   $ ssh <user>@scvbench
   scvbench$ sudo mv benchmark.service /etc/systemd/system/
   scvbench$ sudo systemctl enable --now benchmark

Further steps:

1. Setup chrony (/etc/chrony.conf) to use internal servers

   server 146.107.1.13 trust
   server 146.107.5.10

2. Performance setup

Deployment

1. Make changes in <branch> (either main or a PR branch) and wait until CI finishes.
2. Run nu scripts/deploy.nu <branch> --user=<user>.
3. Trigger a run, e.g. remove and re-add the benchmark label in PR 11.

Development

For local development:

1. Start the server locally
2. use scripts/test.nu to send a payload (check the script for examples for both steps)