Skip to content

Commit

Permalink
Added parameterized benchmarks, convergence detection
Browse files Browse the repository at this point in the history
  • Loading branch information
Izaakwltn committed Oct 31, 2024
1 parent e4303de commit 12ef323
Show file tree
Hide file tree
Showing 6 changed files with 399 additions and 117 deletions.
31 changes: 31 additions & 0 deletions benchmarking/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,37 @@ Benchmarks can be defined in any Coalton package (that imports or nicknames `#:c
Unit)))
```

Parameterized benchmarks allow for multiple inputs to the same function:

```
(define-parameterized-benchmark rec-fib 1000
(fn (x)
(fib x)
Unit)
(seq:make 10 15 20 25))
```
Using `:detect-convergence?` will add standard deviation information to your run, as well as dynamic sample scaling.

### Dynamic sample scaling


With `:detect-convergence?` selected, benchmarks will run one sample at a time, collecting live standard deviation information for timings:

```
(define-parameterized-benchmark rec-fib-generic 1000
(fn (x)
(fib-generic-wrapped x)
Unit)
(seq:make 10 15 20 25)
:detect-convergence? cl:t)
```

When running this mode, the `samples` input is interpreted as the minimum samples, and the benchmark will continue running until the standard deviation of time converges. This ensures the minimum amount of samples before a reliable average.

This mode is most effective on benchmarks with significant individual sample times.

## Running individual benchmarks

Individual benchmarks can be run with `#'run-benchmark`, as long as the benchmark is defined in the current package.
Expand Down
Loading

0 comments on commit 12ef323

Please sign in to comment.