Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: EXC-1818: Add scripts to run all embedder and EE benchmarks #3054

Merged
merged 7 commits into from
Dec 12, 2024

Conversation

berestovskyy
Copy link
Contributor

@berestovskyy berestovskyy commented Dec 9, 2024

This PR adds scripts to run all embedder and execution environment benchmarks. The script to run all benchmarks is run-all-benchmarks.sh, which repeats benchmarks 10 times. The repetition time can be adjusted with the REPEAT environment variable:

REPEAT=3 run-all-benchmarks.sh

The first time the script runs, it quickly executes the benchmarks to identify any broken ones. At the end, the script summarizes the changes. Example run:

==> Summarizing Embedders Heap results:
    MIN_FILE     := EMBEDDERS_HEAP.min
    BASELINE_DIR := ./rs/execution_environment/benches/baseline
= b36fae731e..44e6fe4067: System API Inspect Message total time: 672 ms (improved by -65.2%)
+ inspect/wasm64/ic0_accept_message()* time regressed by 770.6%
- inspect/wasm32/ic0_msg_method_name_copy()/1B time improved by -90%
- inspect/wasm32/ic0_accept_message()* time improved by -19%

@github-actions github-actions bot added the perf label Dec 9, 2024
@berestovskyy berestovskyy marked this pull request as ready for review December 9, 2024 15:24
@berestovskyy berestovskyy requested a review from a team as a code owner December 9, 2024 15:24
@berestovskyy berestovskyy force-pushed the andriy/exc-1818-run-all-benchmarks branch 2 times, most recently from 7b55a14 to 99c7022 Compare December 10, 2024 14:11
@berestovskyy berestovskyy force-pushed the andriy/exc-1818-run-all-benchmarks branch from 99c7022 to 2dc326b Compare December 10, 2024 14:24
Base automatically changed from andriy/exc-1818-fix-system-api-benches to master December 12, 2024 09:29
@berestovskyy berestovskyy added this pull request to the merge queue Dec 12, 2024
Merged via the queue into master with commit 02b2518 Dec 12, 2024
25 checks passed
@berestovskyy berestovskyy deleted the andriy/exc-1818-run-all-benchmarks branch December 12, 2024 13:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants