wasm: Add WAMR support #16057

leyao-daily · 2021-04-19T05:50:57Z

Commit Message: Enabling WAMR(Webassembly Micro Runtime) as wasm runtime in Envoy
Additional Description:

WebAssembly Micro Runtime (WAMR) is a standalone WebAssembly (WASM) runtime with a small footprint. It includes a few parts as below:

The "iwasm" VM core, supporting WebAssembly interpreter, ahead of time compilation (AoT) and Just-in-Time compilation (JIT)

The application framework and the supporting API's for the WASM applications

The dynamic management of the WASM applications

Risk Level: Medium
Testing: Runtime unit testing, integration testing and manual testing

...
//test/integration:http2_flood_integration_test                          PASSED in 13.5s
  Stats over 4 runs: max = 13.5s, min = 11.3s, avg = 12.7s, dev = 0.8s
//test/integration:multiplexed_integration_test                          PASSED in 19.1s
  Stats over 4 runs: max = 19.1s, min = 13.5s, avg = 16.5s, dev = 2.0s
//test/common/http/http2:codec_impl_test                                 PASSED in 4.9s
  Stats over 5 runs: max = 4.9s, min = 3.6s, avg = 4.2s, dev = 0.5s
//test/integration:protocol_integration_test                             PASSED in 40.3s
  Stats over 5 runs: max = 40.3s, min = 32.0s, avg = 35.4s, dev = 2.9s
//test/integration:quic_http_integration_test                            PASSED in 13.0s
  Stats over 6 runs: max = 13.0s, min = 9.6s, avg = 10.7s, dev = 1.2s
//test/integration:quic_protocol_integration_test                        PASSED in 40.3s
  Stats over 8 runs: max = 40.3s, min = 23.5s, avg = 29.4s, dev = 5.4s
//test/extensions/filters/http/wasm:config_test                          PASSED in 3.2s
  Stats over 50 runs: max = 3.2s, min = 0.4s, avg = 0.7s, dev = 0.6s
//test/extensions/filters/http/wasm:wasm_filter_test                     PASSED in 6.2s
  Stats over 50 runs: max = 6.2s, min = 0.9s, avg = 3.5s, dev = 1.2s

Executed 855 out of 855 tests: 855 tests pass.

Docs Changes: N/A
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Deprecated:]
[Optional API Considerations:]

repokitteh-read-only · 2021-04-19T05:51:00Z

Hi @leyao-daily, welcome and thank you for your contribution.

We will try to review your Pull Request as quickly as possible.

In the meantime, please take a look at the contribution guidelines if you have not done so already.

🐱

Caused by: #16057 was opened by leyao-daily.

see: more, trace.

repokitteh-read-only · 2021-04-19T05:51:02Z

CC @envoyproxy/dependency-shepherds: Your approval is needed for changes made to (bazel/.*repos.*\.bzl)|(bazel/dependency_imports\.bzl)|(api/bazel/.*\.bzl)|(.*/requirements\.txt)|(.*\.patch).

🐱

Caused by: #16057 was opened by leyao-daily.

see: more, trace.

bazel/external/wamr.BUILD

bazel/repository_locations.bzl

PiotrSikora

Thank you for working on this. You're still missing:

Wasm Runtime factory for WAMR (see: source/extensions/wasm_runtime),
tests executed on the CI (see: bazel.compile_time_options in ci/do_ci.sh).

test/extensions/bootstrap/wasm/wasm_speed_test.cc

bazel/repository_locations.bzl

repokitteh-read-only · 2021-04-20T08:14:31Z

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to api/envoy/.
envoyproxy/api-shepherds assignee is @markdroth
CC @envoyproxy/api-watchers: FYI only for changes made to api/envoy/.

🐱

Caused by: #16057 was synchronize by leyao-daily.

see: more, trace.

htuch · 2021-04-20T14:28:44Z

bazel/repository_locations.bzl

@@ -704,6 +704,19 @@ REPOSITORY_LOCATIONS_SPEC = dict(
        extensions = ["envoy.wasm.runtime.wavm"],
        cpe = "cpe:2.3:a:webassembly_virtual_machine_project:webassembly_virtual_machine:*",
    ),
+    com_github_wamr = dict(
+        project_name = "Webassembly Micro Runtime",


@PiotrSikora what's our end game here? Will we eventually bring every Wasm runtime into the Envoy OSS build? Do we need this or can we be opinionated?

My main worry here is the increase in build time in CI, security risk envelope, etc.

CC @envoyproxy/dependency-shepherds

Those are good questions.

From Proxy-Wasm C++ Host's perspective, supporting any decent Wasm runtime is desireable, since they all have a bit different features, and most implement variants of Wasm C/C++ API, so both implementation and maintanance have relatively small cost.

From Envoy's perspective, it's definitely not neccessary, and we could be opinionated, but since those are optional alternatives, I don't think there are too many downsides, especially since the common logic is abstracted away.

Regarding build time - Wasm runtimes are updated once a month or once a quarter, and the alternative ones are built only in bazel.compile_time_options target, so their artifacts should be pretty much always cached, and the overhead shouldn't be much of an issue.

Regarding security risk - we could officially support only the default Wasm runtime (currently: V8), and be upfront that there won't be any security releases for other runtimes.

Regarding WAMR specifically, it has interpreted mode, and claims very small binary size overhead and low memory usage, so it might be more desirable in Envoy Mobile than V8, for example.

OK, can we add to this PR a change to threat model https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/threat_model#core-and-extensions stating this explicitly?

I think this is a pretty important thing to get clear as we add more runtimes; we won't support anything other than V8 for now from a security perspective.

htuch · 2021-04-21T18:03:33Z

source/extensions/wasm_runtime/wamr/BUILD

+    name = "config",
+    srcs = ["config.cc"],
+    category = "envoy.wasm.runtime",
+    security_posture = "unknown",


Can we add a comment that "This may never change from unknown until the threat model at https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/threat_model#core-and-extensions is updated to capture additional Wasm runtimes". This should also be done for all other Wasm runtimes other than V8.

So add an annotation above the unknown line to explain the meaning?

Yes. Could you add that comment here and to Wasmtime and WAVM?

Friendly ping.

This is still needed.

leyao-daily · 2021-04-26T02:06:10Z

The PR is waiting for #16112 merging.

PiotrSikora · 2021-04-27T23:51:45Z

The PR is waiting for #16112 merging.

The fix for removed getCustomSection() was merged in #16162, so you should be able to unblock this PR when you update your PR in Proxy-Wasm C++ Host repo, and merge with main branch here.

leyao-daily · 2021-05-13T05:21:44Z

About the API change, what should I do to pass the check?

PiotrSikora · 2021-05-13T06:07:52Z

About the API change, what should I do to pass the check?

Wait for one of the @envoyproxy/api-shepherds to approve.

adisuissa · 2021-05-13T13:21:39Z

/lgtm api

PiotrSikora · 2021-05-19T04:15:02Z

@leyao-daily this needs merge with master to resolve conflicts.

@htuch @lizan could one of you please take a look at this PR?

- generate llvm libraries list with "llvm-config --libnames" ``` $ llvm-config --version 10.0.0 $ llvm-config --libnames asmparser core debuginfodwarf engine lto mcparser mirparser orcjit passes runtimedyld support x86asmparser x86desc ``` - change WAMR default mode to JIT. - change to latest bytecodealliance/wasm-micro-runtime commits Signed-off-by: Liang He <[email protected]>

fix several comments

Signed-off-by: Le Yao <[email protected]>

htuch · 2021-05-21T03:27:16Z

/lgtm deps

Signed-off-by: Le Yao <[email protected]>

leyao-daily · 2021-05-28T08:37:52Z

Once the CI/CD pass, I think this PR is ready to be merged.

source/extensions/extensions_metadata.yaml

Signed-off-by: Le Yao <[email protected]>

htuch

/lgtm deps

WebAssembly Micro Runtime (WAMR) is a standalone WebAssembly (WASM) runtime with a small footprint. It includes a few parts as below: The "iwasm" VM core, supporting WebAssembly interpreter, ahead of time compilation (AoT) and Just-in-Time compilation (JIT) The application framework and the supporting API's for the WASM applications The dynamic management of the WASM applications Risk Level: Medium Testing: Runtime unit testing, integration testing and manual testing Signed-off-by: Le Yao <[email protected]>

leyao-daily requested review from lizan and PiotrSikora as code owners April 19, 2021 05:50

repokitteh-read-only bot added the deps Approval required for changes to Envoy's external dependencies label Apr 19, 2021

lum1n0us reviewed Apr 19, 2021

View reviewed changes

bazel/external/wamr.BUILD Outdated Show resolved Hide resolved

lum1n0us reviewed Apr 19, 2021

View reviewed changes

bazel/repository_locations.bzl Outdated Show resolved Hide resolved

leyao-daily mentioned this pull request Apr 19, 2021

Add support for WebAssembly Micro Runtime (WAMR). proxy-wasm/proxy-wasm-cpp-host#142

Merged

asraa reviewed Apr 19, 2021

View reviewed changes

bazel/repository_locations.bzl Outdated Show resolved Hide resolved

asraa assigned PiotrSikora Apr 19, 2021

PiotrSikora suggested changes Apr 20, 2021

View reviewed changes

test/extensions/bootstrap/wasm/wasm_speed_test.cc Outdated Show resolved Hide resolved

moderation reviewed Apr 20, 2021

View reviewed changes

bazel/repository_locations.bzl Outdated Show resolved Hide resolved

leyao-daily force-pushed the main branch 3 times, most recently from 14643f7 to b2be443 Compare April 20, 2021 05:51

leyao-daily requested a review from PiotrSikora April 20, 2021 05:52

leyao-daily force-pushed the main branch 3 times, most recently from f0752b5 to 49f1de8 Compare April 20, 2021 08:14

repokitteh-read-only bot added the api label Apr 20, 2021

repokitteh-read-only bot assigned markdroth Apr 20, 2021

htuch suggested changes Apr 20, 2021

View reviewed changes

htuch reviewed Apr 21, 2021

View reviewed changes

leyao-daily force-pushed the main branch from 49f1de8 to db37278 Compare April 22, 2021 06:42

leyao-daily mentioned this pull request Apr 22, 2021

To file the addition new dependency to support WAMR #16115

Closed

leyao-daily force-pushed the main branch from db37278 to abde7d6 Compare April 26, 2021 06:18

PiotrSikora mentioned this pull request Apr 26, 2021

docs: Update the document for wasm runtime addition #16137

Closed

repokitteh-read-only bot removed the deps Approval required for changes to Envoy's external dependencies label May 12, 2021

repokitteh-read-only bot removed the api label May 13, 2021

Merge branch 'main' into main

13b52d6

leyao-daily dismissed PiotrSikora’s stale review via 13b52d6 May 20, 2021 05:55

repokitteh-read-only bot added the deps Approval required for changes to Envoy's external dependencies label May 20, 2021

lum1n0us and others added 3 commits May 20, 2021 14:10

Merge pull request #61 from lum1n0us/enable_wamr_jit

6300297

fix several comments

Enable JIT and fix date errors

1eaf38f

Signed-off-by: Le Yao <[email protected]>

repokitteh-read-only bot removed the deps Approval required for changes to Envoy's external dependencies label May 21, 2021

Merge 'envoy/main' into main and fix conflicts

053ae9d

Signed-off-by: Le Yao <[email protected]>

repokitteh-read-only bot added the deps Approval required for changes to Envoy's external dependencies label May 28, 2021

Le Yao added 3 commits May 28, 2021 02:23

Add wamr runtime metadata

b7e910e

Signed-off-by: Le Yao <[email protected]>

Merge branch 'main' into main

5df2800

Conflict fixed based on main

49e1784

Signed-off-by: Le Yao <[email protected]>

htuch reviewed May 28, 2021

View reviewed changes

source/extensions/extensions_metadata.yaml Show resolved Hide resolved

repokitteh-read-only bot added the waiting label May 28, 2021

Add the unknown security posture comments

86420be

Signed-off-by: Le Yao <[email protected]>

repokitteh-read-only bot removed the waiting label May 29, 2021

htuch approved these changes May 31, 2021

View reviewed changes

repokitteh-read-only bot removed the deps Approval required for changes to Envoy's external dependencies label May 31, 2021

htuch merged commit a7fa331 into envoyproxy:main May 31, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

wasm: Add WAMR support #16057

wasm: Add WAMR support #16057

leyao-daily commented Apr 19, 2021

repokitteh-read-only bot commented Apr 19, 2021

repokitteh-read-only bot commented Apr 19, 2021

PiotrSikora left a comment

repokitteh-read-only bot commented Apr 20, 2021

htuch Apr 20, 2021

PiotrSikora Apr 21, 2021

htuch Apr 21, 2021

htuch Apr 21, 2021

leyao-daily Apr 22, 2021

PiotrSikora Apr 27, 2021

PiotrSikora May 6, 2021

PiotrSikora May 7, 2021

leyao-daily commented Apr 26, 2021

PiotrSikora commented Apr 27, 2021

leyao-daily commented May 13, 2021

PiotrSikora commented May 13, 2021

adisuissa commented May 13, 2021

PiotrSikora commented May 19, 2021

htuch commented May 21, 2021

leyao-daily commented May 28, 2021

htuch left a comment

wasm: Add WAMR support #16057

wasm: Add WAMR support #16057

Conversation

leyao-daily commented Apr 19, 2021

repokitteh-read-only bot commented Apr 19, 2021

repokitteh-read-only bot commented Apr 19, 2021

PiotrSikora left a comment

Choose a reason for hiding this comment

repokitteh-read-only bot commented Apr 20, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leyao-daily commented Apr 26, 2021

PiotrSikora commented Apr 27, 2021

leyao-daily commented May 13, 2021

PiotrSikora commented May 13, 2021

adisuissa commented May 13, 2021

PiotrSikora commented May 19, 2021

htuch commented May 21, 2021

leyao-daily commented May 28, 2021

htuch left a comment

Choose a reason for hiding this comment