feat: gnovm benchmarking tool #2241
Conversation
piux2
requested review from
jaekwon,
moul,
deelawn,
zivkovicmilos and
a team
as code owners
github-actions
bot
added
🧾 package/realm
Codecov ReportAttention: Patch coverage is 📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Looks good. Perhaps we could move the transient size variable so that is defined inside the benchmarking package as an exported variable. Then it can be set and read strictly within benchmarking constant conditional blocks and avoid any unnecessary allocations when benchmarking is not enabled.
What do you think?
|
Waiting for updates as per discussions with Ray |
…ive benchmark anymore.
|
Hey @piux2, can you rebase this with |
There was a problem hiding this comment.
The pr provides:
-
Performance measurement capabilities for opcodes (count, execution time) and persistent storage (count, size). This part is similar to performance counter registers in processors. It would better be called
perf(the VM capability) instead ofbenchops(a particular usage of this capability, and not the only possible one, i.e profiling, gas measurement). See for example linux or freebsd -
An executable called
gnobench, to perform a benchmark on a specific scenario (a sample forum board in gno). This executable should not be located under gnovm, but incontribs/benchmark, with
its dependencies:- the readme, makefile, etc in
contrib/benchmark - the executable itself in
contrib/benchmark/cmd/gnobench - benchmarked gno files in
contrib/benchmark/gno - the vm part still in
gnovm/pkg/perf
- the readme, makefile, etc in
Regarding the build tags to enable or not the counting, I'm not a big fan of that. It brings a lot of hassle, and it's partly useless because counting is necessary in production anyway. The performance gain vs using a variable instead of a constant (requiring a different executable) to enable the feature in opcode procressing was shown negligible when we introduced the debugger, it should be the same here. A former colleague of mine used to say: "Premature optimisation is the root of all evil" :-)
In more details, the measurement of opcodes consists of counting and measuring execution time. Counting is super cheap (a counter increment operation), and could be always performed. Measuring execution time is probably a bit more costly and it makes sense to enable it only for benchmark scenarios. In production, we could have a static pre-computed profile of mean exec time per opcode, for gas estimation. But again, no sufficient reason to use build tags.
Regarding the storage of performance counters, they should be stored per virtual machine, instead of a global variable measure, which is generally a bad practice, bringing a lot of unstated constraints and risks. I understand that it is already the case for other parts of the VM, and that it facilitates the coding, but this is something that we should eventually fix. It can be done later, I don't want to block the PR for that.
🛠 PR Checks SummaryAll Automated Checks passed. ✅ Manual Checks (for Reviewers):
Read More🤖 This bot helps streamline PR reviews by verifying automated checks and providing guidance for contributors and reviewers. ✅ Automated Checks (for Contributors):🟢 Maintainers must be able to edit this pull request (more info) ☑️ Contributor Actions:
☑️ Reviewer Actions:
📚 Resources:Debug
|
done |
|
@mvertes thank you for the comments!
Thank you for the reference.! I can see that Linux's perf tool measures software and hardware events in a way similar to how we measure opcode execution and storage access times. However, in our case, we use a set of standardized contracts in benchops/gno to evaluate the execution time of each opcode in the Gno VM runtime. This forms the essence of our benchmarking. To truly implement performance measurement of the VM, similar to what Linux perf does with software and hardware events, we would need to develop a comprehensive new tool. Alternatively, we might reserve the name 'Gno Performance and Profiling Tools' for measuring the VM's execution of user contracts in terms of gas consumption, similar to how the Go runtime measures performance for Go code.
I moved benchops/cmd out of /pkg and placed it in gnovm/cmd/benchops since it is still part of the VM.
I totally agree and advocate "Premature optimization is the root of all evil". All we did was using build flag to set constant flag instead of a variable flag to determine non-production code. Go complier removes entire if block if the condition is a constant value. The constant flag removes the over head 10-18% when we try to capture the accurate value of the operation in nano second with a timer |
<!-- please provide a detailed description of the changes made in this pull request. --> ## Summary: This PR updates the CPU and Store gas based on results from the benchmarking tool:#2241 For CPU gas, the measurement is in nanoseconds per opcode execution. For storage gas, the measurement is in nanoseconds per byte for each type of Gno store access. ## Changes: We moved the gas meter from the underlying store to the upper Gno store to capture accurate resource consumption for VM transactions. At the same time, we retain the original gas store and gas meter for the Auth Keeper to handle regular blockchain transactions that do not necessarily involve the VM. We also updated the gas-wanted in the integration test to reflect actual gas usage. This can serve as a flag to alert us to future changes that might increase gas assumptions. Additional reasons for these changes include: - The Gno VM store processes additional complex encoding and decoding of data structures, rather than simply reading and writing bytes to disk. - For the above reason, we benchmarked gas for store access at the Gno store level. - We want to avoid consuming gas at two points for a single store access during a VM transaction. Here are the diagrams to explain the store access gas before and after changes #### Before:  #### After:  <details><summary>Contributors' checklist...</summary> - [x] Added new tests - [ ] Provided an example (e.g. screenshot) to aid review or the PR is self-explanatory - [ ] Updated the official documentation or not needed - [ ] No breaking changes were made, or a `BREAKING CHANGE: xxx` message was included in the description - [x] Added references to related issues and PRs - [ ] Provided any useful hints for running manual tests </details>
<!-- please provide a detailed description of the changes made in this pull request. --> <details><summary>Contributors' checklist...</summary> - [ x] Added new tests, or not needed, or not feasible - [ x] Provided an example (e.g. screenshot) to aid review or the PR is self-explanatory - [ x] Updated the official documentation or not needed - [ x] No breaking changes were made, or a `BREAKING CHANGE: xxx` message was included in the description - [ x] Added references to related issues and PRs - [ x] Provided any useful hints for running manual tests - [ ] Added new benchmarks to [generated graphs](https://gnoland.github.io/benchmarks), if any. More info [here](https://github.com/gnolang/gno/blob/master/.benchmarks/README.md). </details> We build this tool mainly for the following issues gnolang#1826 gnolang#1828 gnolang#1281 gnolang#1973 We could also use it in the following cases. gnolang#1973 gnolang#2222 ### `gnobench` benchmarks the time consumed for each VM CPU OpCode and persistent access to the store, including marshalling and unmarshalling of realm objects. ## Design consideration ### Minimum Overhead and Footprint - Constant build flags enable benchmarking. - Encode operations and measurements in binary. - Dump to a local file in binary. - No logging, printout, or network access involved. ### Accuracy - Pause the timer for storage access while performing VM opcode benchmarking. - Measure each OpCode execution in nanoseconds. - Store access includes the duration for Amino marshalling and unmarshalling. It is built on top of @deelawn's design and framework with @jaekwon's input. gnolang#2073 ## Usage ### Simple mode The benchmark only involves the GnoVM and the persistent store. It benchmarks the bare minimum components, and the results are isolated from other components. We use standardize gno contract to perform the benchmarking. This mode is the best for benchmarking each major release and/or changes in GnoVM. make opcode make storage ### Production mode It benchmarks the node in the production environment with minimum overhead. We can only benchmark with standardize the contract but also capture the live usage in production environment. It gives us a complete picture of the node perform. 1. Build the production node with benchmarking flags: `go build -tags "benchmarkingstorage benchmarkingops" gno.land/cmd/gnoland` 2. Run the node in the production environment. It will dump benchmark data to a benchmark.bin file. 3. call the realm contracts at `gno.land/r/x/benchmark/opcodes` and `gno.land/r/x/benchmark/storage` 4. Stop the server after the benchmarking session is complete. 5. Run the following command to convert the binary dump: `gnobench -bin path_to_benchmark_bin` it converts the binary dump to results.csv and results_stats.csv. ## Results ( Examples ) The benchmarking results are stored in two files: 1. The raw results are saved in results.csv. | Operation | Elapsed Time | Disk IO Bytes | |-----------------|--------------|---------------| | OpEval | 40333 | 0 | | OpPopBlock | 208 | 0 | | OpHalt | 167 | 0 | | OpEval | 500 | 0 | | OpInterfaceType | 458 | 0 | | OpPopBlock | 166 | 0 | | OpHalt | 125 | 0 | | OpInterfaceType | 21125 | 0 | | OpEval | 541 | 0 | | OpEval | 209 | 0 | | OpInterfaceType | 334 | 0 | 2. The averages and standard deviations are summarized in results_stats.csv. | Operation | Avg Time | Avg Size | Time Std Dev | Count | |----------------|----------|----------|--------------|-------| | OpAdd | 101 | 0 | 45 | 300 | | OpAddAssign | 309 | 0 | 1620 | 100 | | OpArrayLit | 242 | 0 | 170 | 700 | | OpArrayType | 144 | 0 | 100 | 714 | | OpAssign | 136 | 0 | 95 | 2900 | | OpBand | 92 | 0 | 30 | 100 | | OpBandAssign | 127 | 0 | 62 | 100 | | OpBandn | 97 | 0 | 54 | 100 | | OpBandnAssign | 125 | 0 | 113 | 100 | | OpBinary1 | 128 | 0 | 767 | 502 | | OpBody | 127 | 0 | 145 | 13700 | --------- Co-authored-by: Morgan Bazalgette <[email protected]>
Contributors' checklist...
BREAKING CHANGE: xxxmessage was included in the descriptionWe build this tool mainly for the following issues
#1826
#1828
#1281
#1973
We could also use it in the following cases.
#1973
#2222
gnobenchbenchmarks the time consumed for each VM CPU OpCode and persistent access to the store, including marshalling and unmarshalling of realm objects.Design consideration
Minimum Overhead and Footprint
Accuracy
It is built on top of @deelawn's design and framework with @jaekwon's input.
#2073
Usage
Simple mode
The benchmark only involves the GnoVM and the persistent store. It benchmarks the bare minimum components, and the results are isolated from other components. We use standardize gno contract to perform the benchmarking.
This mode is the best for benchmarking each major release and/or changes in GnoVM.
Production mode
It benchmarks the node in the production environment with minimum overhead.
We can only benchmark with standardize the contract but also capture the live usage in production environment.
It gives us a complete picture of the node perform.
go build -tags "benchmarkingstorage benchmarkingops" gno.land/cmd/gnolandRun the node in the production environment. It will dump benchmark data to a benchmark.bin file.
call the realm contracts at
gno.land/r/x/benchmark/opcodesandgno.land/r/x/benchmark/storageStop the server after the benchmarking session is complete.
Run the following command to convert the binary dump:
gnobench -bin path_to_benchmark_binResults ( Examples )
The benchmarking results are stored in two files: