Document the version/commithash/date or each VM used #41

axic · 2020-05-15T00:23:45Z

Can start collecting that here in the issue, but eventually should be documented in the readme or similar.

The hash is the upstream hash the branch is based on.

wasm-micro-runtime (branch: benchmark) 130d7d07d0ee649a38b9b8df7d22404d960b2f09 / Jan 2020 / closest tag: WAMR-02-18-2020 (6 commits behind)
wasm3 (branch: benchmark) 565b585697291a864d127f3cd3c383cebd80ab1e / 30 Dec 2019 / closest tag: v0.4.2 (5 commits behind)
wagon (branch: bench-times) e9f4420cbd6da59babf15527294da199116aa0ff / 22 Feb 2019 / closest tag: 0.4.0 (3 commits behind)
wasmi (branch: bench-time) 23b054c0e5139f0266b6189deb4279749b8058d5 / 20 Jan 2019 / closest tag: 0.4.4 (5 commits behind)

The text was updated successfully, but these errors were encountered:

axic · 2020-05-23T16:16:59Z

Suggestion: rebase these branches on top of the closest release and name them as <version>-benchmarking, e.g. wasm3 would have a branch called v0.4.2-benchmarking based on the v0.4.2 release.

hugo-dc · 2020-05-27T05:38:47Z

Here is the list of versions of VMs used:

go-ethereum
- branch: master
  - commithash: 6d74d1e5f762e06a6a739a42261886510f842778
  - date: May 13, 2020
  - closest tag: v1.9.14
  - proposed branch name: v1.9.14-benchmarking
openethereum
- branch: evm-code-bencher
  - commithash: ecbafb2390b67b7a56902150de8c1cab44eb6fd7
  - date: Jun 11, 2019
  - closest tag: v2.5.2
  - proposed branch name: v2.5.2-benchmarking
evmone
- branch: bench-evm-codes
  - commithash: 94f4e82713c39812ec00a0e229f5906cea609d75
  - date: Jun 19, 2019
  - closest tag: v0.1.0
  - proposed branch name: v0.1.0-benchmarking
cita-vm
- branch: evm-bencher
  - commithash: 6fbc419690d968f2a524b9373f93014e1b942323
  - date: May 21, 2019
  - closest tag: master
  - proposed branch name: 6fbc41-benchmarking
wabt
- branch: bench-times
  - commithash: 04fe0c4124967f09060f3a9d16e2bf4f7956ac0d
  - date: Mar 6
  - closest tag: 1.0.9
  - proposed branch name: 1.0.9-benchmarking
wasmtime
- branch: bench-times
  - commithash: fb7c1b77958c9ae850edd2392c85c83dfe27ed00
  - date: Feb 26, 2019
  - closest tag: 0.31.0 (more than 400 commits)
  - proposed branch name: 0.31.0-benchmarking
asmble
- branch: bench-times
  - commithash: 9172fba94882a08a8a975c219e4afd96f5b01d5e
  - date: Nov 15, 2018
  - closest tag: 0.4.2-fl
  - proposed branch name: 0.4.2-fl-benchmarking
wavm
- branch: bench-compile-time
  - commithash: d3607084b98017243cefe699064d4d5cf3f50fd9
  - date: Feb 14, 2019
  - closest tag: nightly-2019-08-28
  - proposed branch name: nightly-2019-08-28-benchmarking
life
- branch: bench-times
  - commithash: d05763d110500d0a2bf15724c6aee6b225437a87
  - date: Feb 4, 2019
  - closest tag: no-tags
  - proposed branch name: d05763-benchmarking
ssvm
- branch: bench
  - commithash: dc97dd1954581d6ab9966caf25b7baf52c463b39
  - date: Apr 9, 2020
  - closest tag: 0.5.1
  - proposed branch name: 0.5.1-benchmarking
wasm-micro-runtime
- branch: benchmark
  - commithash: 130d7d07d0ee649a38b9b8df7d22404d960b2f09
  - date: Jan 2020
  - closest tag: WAMR-02-18-2020 (6 commits behind)
wasm3
- branch: benchmark
  - commithash: 565b585697291a864d127f3cd3c383cebd80ab1e
  - date: 30 Dec 2019
  - closest tag: v0.4.2 (5 commits behind)
wagon
- branch: bench-times
  - commithash: e9f4420cbd6da59babf15527294da199116aa0ff
  - date: 22 Feb 2019
  - closest tag: 0.4.0 (3 commits behind)
wasmi
- branch: bench-time
  - commithash: 23b054c0e5139f0266b6189deb4279749b8058d5
  - date: 20 Jan 2019
  - closest tag: 0.4.4 (5 commits behind)
rollup.rs
- branch: benchreport-rust-native
  - commithash: a9650ddf54a46ece0a9a661f835cc6b231df1c51
  - date: Aug 28, 2019
  - closest tag: no-tags
  - proposed branch name: a9650d-rust-native-benchmarking
- branch: benchreport-scout-no-bignums
  - commithash: a9650ddf54a46ece0a9a661f835cc6b231df1c51
  - date: Aug 28, 2019
  - closest tag: no-tags
  - proposed branch name: a9650d-scout-no-bignum-benchmarking
- branch: benchreport-scout-bignums
  - commithash: a9650ddf54a46ece0a9a661f835cc6b231df1c51
  - date: Aug 28, 2019
  - closest tag: no-tags
  - proposed branch name: a9650d-scout-bignum-benchmarking
daiquiri
- branch: scout-benchreport-withdraw
  - commithash: d9d047943e3792335b603a08e8dc40fbb8688777
  - date: Mar 3, 2020
  - closest tag: no-tags
  - proposed branch name: d9d047-benchmarking
scout_wabt.cpp
- branch: bignum-host-funcs
  - commithash: 4dda8694a51d77f78397d034abd3203e338523c5
  - date: Dec 17, 2019
  - closest tag: no-tags
  - proposed branch name: 4dda86-bignum-benchmarking
- branch: bignum-hostfuncs-secp256k1
  - commithash: 4dda8694a51d77f78397d034abd3203e338523c5
  - date: Dec 17, 2019
  - closest tag: no-tags
  - proposed branch name: 4dda86-bignum-secp-benchmarking
- branch: bignum-hostfuncs-bn128-rolluprs
  - commithash: 4dda8694a51d77f78397d034abd3203e338523c5
  - date: Dec 17, 2019
  - closest tag: no-tags
  - proposed branch name: 4dda86-bignum-bn128-benchmarking
biturbo
- branch: scout-benchreport
  - commithash: 570634bf12cef824f695c16ecdda37eda866b08d
  - date: Jan 13, 2020
  - closest tag: no-tags
  - proposed branch name: 570634-benchmarking
scout.ts
- branch: secp-sig-verify
  - commithash: 2742ec39a913702c5942840340d84dda27e81fbe
  - date: Oct 1, 2019
  - closest tag: v0.0.2
  - proposed branch name: v0.0.2-secp-benchmarking
- branch: bn127-pairing
  - commithash: 2742ec39a913702c5942840340d84dda27e81fbe
  - date: Oct 1, 2019
  - closest tag: v0.0.2
  - proposed branch name: v0.0.2-bn128-benchmarking
c_ewasm_contracts
- branch: master
  - commithash: 29356a22ea38cf2f1bbab05a4b8974822834f4e7
  - date: Jan 28, 2020
  - closest tag: no-tags
  - proposed branch name: 29356a-benchmarking

axic · 2020-06-07T14:40:53Z

@jwasinger @hugo-dc rebased the wasmi commit on wasmi 0.4.3 and moved the patch from examples/invoke.rs to src/bin/invoke.rs. This means it can be built with cargo build --release without requiring the tests and wabt to build...

It is available at https://github.com/ewasm-benchmarking/wasmi/tree/v0.4.3-benchmarking

jwasinger · 2020-06-08T04:25:26Z

btw I tried that wasmi branch and I get an error:

vms_to_run: {'wasmi': <project.VMDescriptor.VMDescriptor object at 0x7fecf4d629d0>}
06/08/2020 04:23:45 AM <wasm_bencher>: launch guido-fuzzer-find-2 test
06/08/2020 04:23:45 AM <wasm_bencher>: /engines/wasmi/invoke /wasmfilesminified/guido-fuzzer-find-2.wasm main
['/engines/wasmi/invoke', '/wasmfilesminified/guido-fuzzer-find-2.wasm', 'main']
stdout_lines:  thread 'main' panicked at 'index out of bounds: the len is 0 but the index is 0', examples/invoke.rs:81:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
compile_lines:  thread 'main' panicked at 'index out of bounds: the len is 0 but the index is 0', examples/invoke.rs:81:21
compile_match:  None
06/08/2020 04:23:45 AM <wasm_bencher>: wasmi (/engines/wasmi/invoke /wasmfilesminified/guido-fuzzer-find-2.wasm main) got exception:
Error parsing engine output. exception: 'NoneType' object is not subscriptable
engine output:
thread 'main' panicked at 'index out of bounds: the len is 0 but the index is 0', examples/invoke.rs:81:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
06/08/2020 04:23:45 AM <wasm_bencher>: skipping engine wasmi for bench input guido-fuzzer-find-2
06/08/2020 04:23:45 AM <wasm_bencher>: launch bls12-381-aggreg-64-pubkeys-verify-sig test
06/08/2020 04:23:45 AM <wasm_bencher>: /engines/wasmi/invoke /wasmfilesminified/bls12-381-aggreg-64-pubkeys-verify-sig.wasm main

axic · 2020-06-09T22:04:07Z

Also pushed wasm3 v0.4.7-benchmarking: https://github.com/ewasm-benchmarking/wasm3/tree/v0.4.7-benchmarking

axic · 2020-06-09T22:07:01Z

btw I tried that wasmi branch and I get an error:

Weird. We can try based those commits on v0.4.4: https://github.com/ewasm-benchmarking/wasmi/tree/v0.4.4-benchmarking

axic · 2020-06-09T23:53:26Z

We should also use git tags and not git branches on ewasm-benchmarking/<repo>.

The reason: those can not be changed (well, not as easily as branches), so the benchmarking is way more reproducible.

axic · 2020-06-12T18:44:21Z

New suggestion:

Keep a benchmarking branch on each of the forks with the code.
Tag each release, i.e.v1.2.3-benchmarking.

The reason: the tags can't be modified easily, but the branches are easily noticeable if we want to work on them later.

axic · 2020-06-22T17:05:36Z

Updates:

Most of the easy pickings and/or useful ones were done: ssvm, wasm3, evmone, cita-vm, and parity
Two are still pending: wasmi and wagon
The following I don't think is worth bothering with given it is a lot of commits apart: wavm
We are not touching wabt, but rather renaming it, and adding vanilla-wabt (some recent version)
Still need to investigate: wamr and wasmtime

axic · 2020-06-26T15:59:26Z

I tried figuring out the a closeby release for wasmtime but I think it is just too far apart. Would be nice to use a newer wasmtime, but for now I suggest we leave it alone.

Closing this and opening a tracking issue for wasmtime.

axic assigned hugo-dc Jun 7, 2020

This was referenced Jun 9, 2020

Use wasmi 0.4.4 and remove the need for llvm #59

Merged

Use wasm3 0.4.7 #60

Merged

Document versions in the readme #65

Merged

Use ssvm 0.6.0 #66

Merged

axic mentioned this issue Jun 12, 2020

Change cita-vm to v0.1.3 #73

Merged

axic mentioned this issue Jun 12, 2020

Change parity to v2.5.1 #74

Merged

axic mentioned this issue Jun 22, 2020

Upgrade wagon to 0.4.0 #100

Merged

hugo-dc mentioned this issue Jun 26, 2020

Wamr 02 18 2020 #117

Merged

axic closed this as completed Jun 26, 2020

axic mentioned this issue Jun 26, 2020

Use a newer wasmtime #118

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document the version/commithash/date or each VM used #41

Document the version/commithash/date or each VM used #41

axic commented May 15, 2020 •

edited

Loading

axic commented May 23, 2020

hugo-dc commented May 27, 2020

axic commented Jun 7, 2020 •

edited

Loading

jwasinger commented Jun 8, 2020

axic commented Jun 9, 2020

axic commented Jun 9, 2020

axic commented Jun 9, 2020

axic commented Jun 12, 2020

axic commented Jun 22, 2020

axic commented Jun 26, 2020

Document the version/commithash/date or each VM used #41

Document the version/commithash/date or each VM used #41

Comments

axic commented May 15, 2020 • edited Loading

axic commented May 23, 2020

hugo-dc commented May 27, 2020

axic commented Jun 7, 2020 • edited Loading

jwasinger commented Jun 8, 2020

axic commented Jun 9, 2020

axic commented Jun 9, 2020

axic commented Jun 9, 2020

axic commented Jun 12, 2020

axic commented Jun 22, 2020

axic commented Jun 26, 2020

axic commented May 15, 2020 •

edited

Loading

axic commented Jun 7, 2020 •

edited

Loading