Add support for the Component Model Async ABI #38

dicej · 2024-10-08T23:06:07Z

Add support for the Component Model Async ABI to wasm-tools, wit-bindgen, and wasmtime.

Add support for the Component Model [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) to `wasm-tools`, `wit-bindgen`, and `wasmtime`. [Rendered RFC](https://github.com/dicej/rfcs/blob/component-async/accepted/component-model-async.md) Signed-off-by: Joel Dice <[email protected]>

This adds support for encoding and parsing components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) and associated canonical options and functions, along with the [`stream`, `future`, and `error`](WebAssembly/component-model#405) types. Note that the `error` type was recently (about 30 minutes ago) renamed to `error-context` in Luke's spec PR. I haven't updated this implementation to reflect that yet, but will do so in a follow-up commit. That should allow us to avoid conflicts with existing WIT files that use `error` as a type and/or interface name. This does not include any new tests; I'll also add those in a follow-up commit. See bytecodealliance/rfcs#38 for more context. Signed-off-by: Joel Dice <[email protected]>

This adds support for loading, compiling, linking, and running components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) along with the [`stream`, `future`, and `error-context`](WebAssembly/component-model#405) types. It also adds support for generating host bindings such that multiple host functions can be run concurrently with guest tasks -- without monopolizing the `Store`. See the [implementation RFC](bytecodealliance/rfcs#38) for details, as well as [this repo](https://github.com/dicej/component-async-demo) containing end-to-end smoke tests. This is very much a work-in progress, with a number of tasks remaining: - [ ] Avoid exposing global task IDs to guests and use per-instance IDs instead - [ ] Track `task.return` type during compilation and assert the actual and expected types match at runtime - [ ] Ensure all guest pointers are bounds-checked when lifting, lowering, or copying values - [ ] Reduce code duplication in `wasmtime_cranelift::compiler::component` - [ ] Add support for `(Typed)Func::call_concurrent` per the RFC - [ ] Add support for multiplexing stream/future reads/writes and concurrent calls to guest exports per the RFC - [ ] Refactor, clean up, and unify handling of backpressure, yields, and even polling - [ ] Guard against reentrance where required (e.g. in certain fused adapter calls) - [ ] Add integration test cases covering new functionality to tests/all/component_model (starting by porting over the tests in https://github.com/dicej/component-async-demo) - [ ] Add binding generation test cases to crates/component-macro/tests - [ ] Add WAST tests to tests/misc_testsuite/component-model - [ ] Add support and test coverage for callback-less async functions (e.g. goroutines) - [ ] Switch to back to upstream `wasm-tools` once bytecodealliance/wasm-tools#1895 has been merged and released Signed-off-by: Joel Dice <[email protected]>

This adds support for loading, compiling, linking, and running components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) along with the [`stream`, `future`, and `error-context`](WebAssembly/component-model#405) types. It also adds support for generating host bindings such that multiple host functions can be run concurrently with guest tasks -- without monopolizing the `Store`. See the [implementation RFC](bytecodealliance/rfcs#38) for details, as well as [this repo](https://github.com/dicej/component-async-demo) containing end-to-end smoke tests. This is very much a work-in progress, with a number of tasks remaining: - [ ] Avoid exposing global task IDs to guests and use per-instance IDs instead - [ ] Track `task.return` type during compilation and assert the actual and expected types match at runtime - [ ] Ensure all guest pointers are bounds-checked when lifting, lowering, or copying values - [ ] Reduce code duplication in `wasmtime_cranelift::compiler::component` - [ ] Reduce code duplication between `StoreContextMut::on_fiber` and `concurrent::on_fiber` - [ ] Minimize and/or document the use of unsafe code - [ ] Add support for `(Typed)Func::call_concurrent` per the RFC - [ ] Add support for multiplexing stream/future reads/writes and concurrent calls to guest exports per the RFC - [ ] Refactor, clean up, and unify handling of backpressure, yields, and even polling - [ ] Guard against reentrance where required (e.g. in certain fused adapter calls) - [ ] Add integration test cases covering new functionality to tests/all/component_model (starting by porting over the tests in https://github.com/dicej/component-async-demo) - [ ] Add binding generation test cases to crates/component-macro/tests - [ ] Add WAST tests to tests/misc_testsuite/component-model - [ ] Add support and test coverage for callback-less async functions (e.g. goroutines) - [ ] Switch to back to upstream `wasm-tools` once bytecodealliance/wasm-tools#1895 has been merged and released Signed-off-by: Joel Dice <[email protected]>

This adds support for encoding and parsing components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) and associated canonical options and functions, along with the [`stream`, `future`, and `error`](WebAssembly/component-model#405) types. Note that the `error` type was recently (about 30 minutes ago) renamed to `error-context` in Luke's spec PR. I haven't updated this implementation to reflect that yet, but will do so in a follow-up commit. That should allow us to avoid conflicts with existing WIT files that use `error` as a type and/or interface name. This does not include any new tests; I'll also add those in a follow-up commit. See bytecodealliance/rfcs#38 for more context. Signed-off-by: Joel Dice <[email protected]> add wasmparser::WasmFeatures support to wasm-compose Signed-off-by: Joel Dice <[email protected]> fix no-std build in readers.rs Signed-off-by: Joel Dice <[email protected]> rename `error` to `error-context` per latest spec Signed-off-by: Joel Dice <[email protected]> rename `error` to `error-context` per latest spec (part 2) Also, parse string encoding and realloc from encoded `error-context.new` and `error-context.debug-string` names. Signed-off-by: Joel Dice <[email protected]> add `wast` support for parsing async canon opts And add tests/local/component-model-async/lift-async.wast for round-trip testing of async lifts (more to come). Signed-off-by: Joel Dice <[email protected]> more wast async support and more tests This also fixes a bug in `wasmprinter` keeping track of core functions. Signed-off-by: Joel Dice <[email protected]> more wast async support; add async tests; fix bugs Signed-off-by: Joel Dice <[email protected]> more component-model-async tests and fixes Signed-off-by: Joel Dice <[email protected]> add `wit-parser` tests for streams, futures, and error-contexts Signed-off-by: Joel Dice <[email protected]> add first `wit-component` async test This required adding a new `wit_parser::decoding::decode_reader_with_features` function for passing `WasmFeatures` to `wasmparser::Validator::new_with_features`. Signed-off-by: Joel Dice <[email protected]> add more async tests Signed-off-by: Joel Dice <[email protected]> add `async-builtins` test for `wit-component` Signed-off-by: Joel Dice <[email protected]> add `async-streams-and-futures` test to `wit-component` Signed-off-by: Joel Dice <[email protected]>

This adds support for loading, compiling, linking, and running components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) along with the [`stream`, `future`, and `error-context`](WebAssembly/component-model#405) types. It also adds support for generating host bindings such that multiple host functions can be run concurrently with guest tasks -- without monopolizing the `Store`. See the [implementation RFC](bytecodealliance/rfcs#38) for details, as well as [this repo](https://github.com/dicej/component-async-demo) containing end-to-end smoke tests. This is very much a work-in progress, with a number of tasks remaining: - [ ] Avoid exposing global task IDs to guests and use per-instance IDs instead - [ ] Track `task.return` type during compilation and assert the actual and expected types match at runtime - [ ] Ensure all guest pointers are bounds-checked when lifting, lowering, or copying values - [ ] Reduce code duplication in `wasmtime_cranelift::compiler::component` - [ ] Reduce code duplication between `StoreContextMut::on_fiber` and `concurrent::on_fiber` - [ ] Minimize and/or document the use of unsafe code - [ ] Add support for `(Typed)Func::call_concurrent` per the RFC - [ ] Add support for multiplexing stream/future reads/writes and concurrent calls to guest exports per the RFC - [ ] Refactor, clean up, and unify handling of backpressure, yields, and even polling - [ ] Guard against reentrance where required (e.g. in certain fused adapter calls) - [ ] Add integration test cases covering new functionality to tests/all/component_model (starting by porting over the tests in https://github.com/dicej/component-async-demo) - [ ] Add binding generation test cases to crates/component-macro/tests - [ ] Add WAST tests to tests/misc_testsuite/component-model - [ ] Add support and test coverage for callback-less async functions (e.g. goroutines) - [ ] Switch to back to upstream `wasm-tools` once bytecodealliance/wasm-tools#1895 has been merged and released Signed-off-by: Joel Dice <[email protected]> fix clippy warnings and bench/fuzzing errors Signed-off-by: Joel Dice <[email protected]> revert atomic.wit whitespace change Signed-off-by: Joel Dice <[email protected]> fix build when component-model disabled Signed-off-by: Joel Dice <[email protected]> bless component-macro expected output Signed-off-by: Joel Dice <[email protected]> fix no-std build error Signed-off-by: Joel Dice <[email protected]> fix build with --no-default-features --features runtime,component-model Signed-off-by: Joel Dice <[email protected]> partly fix no-std build It's still broken due to the use of `std::collections::HashMap` in crates/wasmtime/src/runtime/vm/component.rs. I'll address that as part of the work to avoid exposing global task/future/stream/error-context handles to guests. Signed-off-by: Joel Dice <[email protected]> maintain per-instance tables for futures, streams, and error-contexts Signed-off-by: Joel Dice <[email protected]>

This adds support for encoding and parsing components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) and associated canonical options and functions, along with the [`stream`, `future`, and `error`](WebAssembly/component-model#405) types. Note that the `error` type was recently (about 30 minutes ago) renamed to `error-context` in Luke's spec PR. I haven't updated this implementation to reflect that yet, but will do so in a follow-up commit. That should allow us to avoid conflicts with existing WIT files that use `error` as a type and/or interface name. This does not include any new tests; I'll also add those in a follow-up commit. See bytecodealliance/rfcs#38 for more context. Signed-off-by: Joel Dice <[email protected]> add wasmparser::WasmFeatures support to wasm-compose Signed-off-by: Joel Dice <[email protected]> fix no-std build in readers.rs Signed-off-by: Joel Dice <[email protected]> rename `error` to `error-context` per latest spec Signed-off-by: Joel Dice <[email protected]> rename `error` to `error-context` per latest spec (part 2) Also, parse string encoding and realloc from encoded `error-context.new` and `error-context.debug-string` names. Signed-off-by: Joel Dice <[email protected]> add `wast` support for parsing async canon opts And add tests/local/component-model-async/lift-async.wast for round-trip testing of async lifts (more to come). Signed-off-by: Joel Dice <[email protected]> more wast async support and more tests This also fixes a bug in `wasmprinter` keeping track of core functions. Signed-off-by: Joel Dice <[email protected]> more wast async support; add async tests; fix bugs Signed-off-by: Joel Dice <[email protected]> more component-model-async tests and fixes Signed-off-by: Joel Dice <[email protected]> add `wit-parser` tests for streams, futures, and error-contexts Signed-off-by: Joel Dice <[email protected]> add first `wit-component` async test This required adding a new `wit_parser::decoding::decode_reader_with_features` function for passing `WasmFeatures` to `wasmparser::Validator::new_with_features`. Signed-off-by: Joel Dice <[email protected]> add more async tests Signed-off-by: Joel Dice <[email protected]> add `async-builtins` test for `wit-component` Signed-off-by: Joel Dice <[email protected]> add `async-streams-and-futures` test to `wit-component` Signed-off-by: Joel Dice <[email protected]>

This adds support for loading, compiling, linking, and running components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) along with the [`stream`, `future`, and `error-context`](WebAssembly/component-model#405) types. It also adds support for generating host bindings such that multiple host functions can be run concurrently with guest tasks -- without monopolizing the `Store`. See the [implementation RFC](bytecodealliance/rfcs#38) for details, as well as [this repo](https://github.com/dicej/component-async-demo) containing end-to-end smoke tests. This is very much a work-in progress, with a number of tasks remaining: - [ ] Avoid exposing global task IDs to guests and use per-instance IDs instead - [ ] Track `task.return` type during compilation and assert the actual and expected types match at runtime - [ ] Ensure all guest pointers are bounds-checked when lifting, lowering, or copying values - [ ] Reduce code duplication in `wasmtime_cranelift::compiler::component` - [ ] Reduce code duplication between `StoreContextMut::on_fiber` and `concurrent::on_fiber` - [ ] Minimize and/or document the use of unsafe code - [ ] Add support for `(Typed)Func::call_concurrent` per the RFC - [ ] Add support for multiplexing stream/future reads/writes and concurrent calls to guest exports per the RFC - [ ] Refactor, clean up, and unify handling of backpressure, yields, and even polling - [ ] Guard against reentrance where required (e.g. in certain fused adapter calls) - [ ] Add integration test cases covering new functionality to tests/all/component_model (starting by porting over the tests in https://github.com/dicej/component-async-demo) - [ ] Add binding generation test cases to crates/component-macro/tests - [ ] Add WAST tests to tests/misc_testsuite/component-model - [ ] Add support and test coverage for callback-less async functions (e.g. goroutines) - [ ] Switch to back to upstream `wasm-tools` once bytecodealliance/wasm-tools#1895 has been merged and released Signed-off-by: Joel Dice <[email protected]> fix clippy warnings and bench/fuzzing errors Signed-off-by: Joel Dice <[email protected]> revert atomic.wit whitespace change Signed-off-by: Joel Dice <[email protected]> fix build when component-model disabled Signed-off-by: Joel Dice <[email protected]> bless component-macro expected output Signed-off-by: Joel Dice <[email protected]> fix no-std build error Signed-off-by: Joel Dice <[email protected]> fix build with --no-default-features --features runtime,component-model Signed-off-by: Joel Dice <[email protected]> partly fix no-std build It's still broken due to the use of `std::collections::HashMap` in crates/wasmtime/src/runtime/vm/component.rs. I'll address that as part of the work to avoid exposing global task/future/stream/error-context handles to guests. Signed-off-by: Joel Dice <[email protected]> maintain per-instance tables for futures, streams, and error-contexts Signed-off-by: Joel Dice <[email protected]> refactor task/stream/future handle lifting/lowering This addresses a couple of issues: - Previously, we were passing task/stream/future/error-context reps directly to instances while keeping track of which instance had access to which rep. That worked fine in that there was no way to forge access to inaccessible reps, but it leaked information about what other instances were doing. Now we maintain per-instance waitable and error-context tables which map the reps to and from the handles which the instance sees. - The `no_std` build was broken due to use of `HashMap` in `runtime::vm::component`, which is now fixed. Note that we use one single table per instance for all tasks, streams, and futures. This is partly necessary because, when async events are delivered to the guest, it wouldn't have enough context to know which stream or future we're talking about if each unique stream and future type had its own table. So at minimum, we need to use the same table for all streams (regardless of payload type), and likewise for futures. Also, per WebAssembly/component-model#395 (comment), the plan is to move towards a shared table for all resource types as well, so this moves us in that direction. Signed-off-by: Joel Dice <[email protected]> fix wave breakage due to new stream/future/error-context types Signed-off-by: Joel Dice <[email protected]>

This adds support for encoding and parsing components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) and associated canonical options and functions, along with the [`stream`, `future`, and `error`](WebAssembly/component-model#405) types. See bytecodealliance/rfcs#38 for more context. Signed-off-by: Joel Dice <[email protected]> add wasmparser::WasmFeatures support to wasm-compose Signed-off-by: Joel Dice <[email protected]> fix no-std build in readers.rs Signed-off-by: Joel Dice <[email protected]> rename `error` to `error-context` per latest spec Signed-off-by: Joel Dice <[email protected]> rename `error` to `error-context` per latest spec (part 2) Also, parse string encoding and realloc from encoded `error-context.new` and `error-context.debug-string` names. Signed-off-by: Joel Dice <[email protected]> add `wast` support for parsing async canon opts And add tests/local/component-model-async/lift-async.wast for round-trip testing of async lifts (more to come). Signed-off-by: Joel Dice <[email protected]> more wast async support and more tests This also fixes a bug in `wasmprinter` keeping track of core functions. Signed-off-by: Joel Dice <[email protected]> more wast async support; add async tests; fix bugs Signed-off-by: Joel Dice <[email protected]> more component-model-async tests and fixes Signed-off-by: Joel Dice <[email protected]> add `wit-parser` tests for streams, futures, and error-contexts Signed-off-by: Joel Dice <[email protected]> add first `wit-component` async test This required adding a new `wit_parser::decoding::decode_reader_with_features` function for passing `WasmFeatures` to `wasmparser::Validator::new_with_features`. Signed-off-by: Joel Dice <[email protected]> add more async tests Signed-off-by: Joel Dice <[email protected]> add `async-builtins` test for `wit-component` Signed-off-by: Joel Dice <[email protected]> add `async-streams-and-futures` test to `wit-component` Signed-off-by: Joel Dice <[email protected]> fix stream/future type handling for exported interfaces Signed-off-by: Joel Dice <[email protected]> support callback-less (AKA stackful) async lifts Signed-off-by: Joel Dice <[email protected]>

This adds support for loading, compiling, linking, and running components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) along with the [`stream`, `future`, and `error-context`](WebAssembly/component-model#405) types. It also adds support for generating host bindings such that multiple host functions can be run concurrently with guest tasks -- without monopolizing the `Store`. See the [implementation RFC](bytecodealliance/rfcs#38) for details, as well as [this repo](https://github.com/dicej/component-async-demo) containing end-to-end smoke tests. Signed-off-by: Joel Dice <[email protected]> fix clippy warnings and bench/fuzzing errors Signed-off-by: Joel Dice <[email protected]> revert atomic.wit whitespace change Signed-off-by: Joel Dice <[email protected]> fix build when component-model disabled Signed-off-by: Joel Dice <[email protected]> bless component-macro expected output Signed-off-by: Joel Dice <[email protected]> fix no-std build error Signed-off-by: Joel Dice <[email protected]> fix build with --no-default-features --features runtime,component-model Signed-off-by: Joel Dice <[email protected]> partly fix no-std build It's still broken due to the use of `std::collections::HashMap` in crates/wasmtime/src/runtime/vm/component.rs. I'll address that as part of the work to avoid exposing global task/future/stream/error-context handles to guests. Signed-off-by: Joel Dice <[email protected]> maintain per-instance tables for futures, streams, and error-contexts Signed-off-by: Joel Dice <[email protected]> refactor task/stream/future handle lifting/lowering This addresses a couple of issues: - Previously, we were passing task/stream/future/error-context reps directly to instances while keeping track of which instance had access to which rep. That worked fine in that there was no way to forge access to inaccessible reps, but it leaked information about what other instances were doing. Now we maintain per-instance waitable and error-context tables which map the reps to and from the handles which the instance sees. - The `no_std` build was broken due to use of `HashMap` in `runtime::vm::component`, which is now fixed. Note that we use one single table per instance for all tasks, streams, and futures. This is partly necessary because, when async events are delivered to the guest, it wouldn't have enough context to know which stream or future we're talking about if each unique stream and future type had its own table. So at minimum, we need to use the same table for all streams (regardless of payload type), and likewise for futures. Also, per WebAssembly/component-model#395 (comment), the plan is to move towards a shared table for all resource types as well, so this moves us in that direction. Signed-off-by: Joel Dice <[email protected]> fix wave breakage due to new stream/future/error-context types Signed-off-by: Joel Dice <[email protected]> switch wasm-tools to v1.220.0-based branch Signed-off-by: Joel Dice <[email protected]> check `task.return` type at runtime We can't statically verify a given call to `task.return` corresponds to the expected core signature appropriate for the currently running task, so we must do so at runtime. In order to make that check efficient, we intern the types. My initial plan was to use `ModuleInternedTypeIndex` and/or `VMSharedTypeIndex` for interning, but that got hairy with WasmGC considerations, so instead I added new fields to `ComponentTypes` and `ComponentTypesBuilder`. Signed-off-by: Joel Dice <[email protected]> add `TypedFunc::call_concurrent` and refine stream/future APIs This implements what I proposed in https://github.com/dicej/rfcs/blob/component-async/accepted/component-model-async.md#wasmtime. Specifically, it adds: - A new `Promise` type, useful for working with concurrent operations that require access to a `Store` to make progress. - A new `PromisesUnordered` type for `await`ing multiple promises concurrently -`TypedFunc::call_concurrent` (which returns a `Promise`), allowing multiple host->guest calls to run concurrently on the same instance. - Updated `{Stream|Future}{Writer|Reader}` APIs which use `Promise` The upshot is that the embedder can now ergonomically manage arbitrary numbers of concurrent operations. Previously, this was a lot more difficult to do without accidentally starving some of the operations due to another one monopolizing the `Store`. Finally, this includes various refactorings and fixes for bugs exposed by the newer, more versatile APIs. Signed-off-by: Joel Dice <[email protected]> clean up verbosity in component/func.rs Signed-off-by: Joel Dice <[email protected]> snapshot Signed-off-by: Joel Dice <[email protected]> implement stream/future read/write cancellation This required a somewhat viral addition of `Send` and `Sync` bounds for async host function closure types, unfortunately. Signed-off-by: Joel Dice <[email protected]> add `Func::call_concurrent` and `LinkerInstance::func_new_concurrent` Signed-off-by: Joel Dice <[email protected]> dynamic API support for streams/futures/error-contexts Signed-off-by: Joel Dice <[email protected]> support callback-less (AKA stackful) async lifts Signed-off-by: Joel Dice <[email protected]>

This adds support for encoding and parsing components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) and associated canonical options and functions, along with the [`stream`, `future`, and `error`](WebAssembly/component-model#405) types. See bytecodealliance/rfcs#38 for more context. Signed-off-by: Joel Dice <[email protected]> add wasmparser::WasmFeatures support to wasm-compose Signed-off-by: Joel Dice <[email protected]> fix no-std build in readers.rs Signed-off-by: Joel Dice <[email protected]> rename `error` to `error-context` per latest spec Signed-off-by: Joel Dice <[email protected]> rename `error` to `error-context` per latest spec (part 2) Also, parse string encoding and realloc from encoded `error-context.new` and `error-context.debug-string` names. Signed-off-by: Joel Dice <[email protected]> add `wast` support for parsing async canon opts And add tests/local/component-model-async/lift-async.wast for round-trip testing of async lifts (more to come). Signed-off-by: Joel Dice <[email protected]> more wast async support and more tests This also fixes a bug in `wasmprinter` keeping track of core functions. Signed-off-by: Joel Dice <[email protected]> more wast async support; add async tests; fix bugs Signed-off-by: Joel Dice <[email protected]> more component-model-async tests and fixes Signed-off-by: Joel Dice <[email protected]> add `wit-parser` tests for streams, futures, and error-contexts Signed-off-by: Joel Dice <[email protected]> add first `wit-component` async test This required adding a new `wit_parser::decoding::decode_reader_with_features` function for passing `WasmFeatures` to `wasmparser::Validator::new_with_features`. Signed-off-by: Joel Dice <[email protected]> add more async tests Signed-off-by: Joel Dice <[email protected]> add `async-builtins` test for `wit-component` Signed-off-by: Joel Dice <[email protected]> add `async-streams-and-futures` test to `wit-component` Signed-off-by: Joel Dice <[email protected]> fix stream/future type handling for exported interfaces Signed-off-by: Joel Dice <[email protected]> support callback-less (AKA stackful) async lifts Signed-off-by: Joel Dice <[email protected]>

This adds support for loading, compiling, linking, and running components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) along with the [`stream`, `future`, and `error-context`](WebAssembly/component-model#405) types. It also adds support for generating host bindings such that multiple host functions can be run concurrently with guest tasks -- without monopolizing the `Store`. See the [implementation RFC](bytecodealliance/rfcs#38) for details, as well as [this repo](https://github.com/dicej/component-async-demo) containing end-to-end smoke tests. Signed-off-by: Joel Dice <[email protected]> fix clippy warnings and bench/fuzzing errors Signed-off-by: Joel Dice <[email protected]> revert atomic.wit whitespace change Signed-off-by: Joel Dice <[email protected]> fix build when component-model disabled Signed-off-by: Joel Dice <[email protected]> bless component-macro expected output Signed-off-by: Joel Dice <[email protected]> fix no-std build error Signed-off-by: Joel Dice <[email protected]> fix build with --no-default-features --features runtime,component-model Signed-off-by: Joel Dice <[email protected]> partly fix no-std build It's still broken due to the use of `std::collections::HashMap` in crates/wasmtime/src/runtime/vm/component.rs. I'll address that as part of the work to avoid exposing global task/future/stream/error-context handles to guests. Signed-off-by: Joel Dice <[email protected]> maintain per-instance tables for futures, streams, and error-contexts Signed-off-by: Joel Dice <[email protected]> refactor task/stream/future handle lifting/lowering This addresses a couple of issues: - Previously, we were passing task/stream/future/error-context reps directly to instances while keeping track of which instance had access to which rep. That worked fine in that there was no way to forge access to inaccessible reps, but it leaked information about what other instances were doing. Now we maintain per-instance waitable and error-context tables which map the reps to and from the handles which the instance sees. - The `no_std` build was broken due to use of `HashMap` in `runtime::vm::component`, which is now fixed. Note that we use one single table per instance for all tasks, streams, and futures. This is partly necessary because, when async events are delivered to the guest, it wouldn't have enough context to know which stream or future we're talking about if each unique stream and future type had its own table. So at minimum, we need to use the same table for all streams (regardless of payload type), and likewise for futures. Also, per WebAssembly/component-model#395 (comment), the plan is to move towards a shared table for all resource types as well, so this moves us in that direction. Signed-off-by: Joel Dice <[email protected]> fix wave breakage due to new stream/future/error-context types Signed-off-by: Joel Dice <[email protected]> switch wasm-tools to v1.220.0-based branch Signed-off-by: Joel Dice <[email protected]> check `task.return` type at runtime We can't statically verify a given call to `task.return` corresponds to the expected core signature appropriate for the currently running task, so we must do so at runtime. In order to make that check efficient, we intern the types. My initial plan was to use `ModuleInternedTypeIndex` and/or `VMSharedTypeIndex` for interning, but that got hairy with WasmGC considerations, so instead I added new fields to `ComponentTypes` and `ComponentTypesBuilder`. Signed-off-by: Joel Dice <[email protected]> add `TypedFunc::call_concurrent` and refine stream/future APIs This implements what I proposed in https://github.com/dicej/rfcs/blob/component-async/accepted/component-model-async.md#wasmtime. Specifically, it adds: - A new `Promise` type, useful for working with concurrent operations that require access to a `Store` to make progress. - A new `PromisesUnordered` type for `await`ing multiple promises concurrently -`TypedFunc::call_concurrent` (which returns a `Promise`), allowing multiple host->guest calls to run concurrently on the same instance. - Updated `{Stream|Future}{Writer|Reader}` APIs which use `Promise` The upshot is that the embedder can now ergonomically manage arbitrary numbers of concurrent operations. Previously, this was a lot more difficult to do without accidentally starving some of the operations due to another one monopolizing the `Store`. Finally, this includes various refactorings and fixes for bugs exposed by the newer, more versatile APIs. Signed-off-by: Joel Dice <[email protected]> clean up verbosity in component/func.rs Signed-off-by: Joel Dice <[email protected]> snapshot Signed-off-by: Joel Dice <[email protected]> implement stream/future read/write cancellation This required a somewhat viral addition of `Send` and `Sync` bounds for async host function closure types, unfortunately. Signed-off-by: Joel Dice <[email protected]> add `Func::call_concurrent` and `LinkerInstance::func_new_concurrent` Signed-off-by: Joel Dice <[email protected]> dynamic API support for streams/futures/error-contexts Signed-off-by: Joel Dice <[email protected]> support callback-less (AKA stackful) async lifts Signed-off-by: Joel Dice <[email protected]>

This adds support for encoding and parsing components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) and associated canonical options and functions, along with the [`stream`, `future`, and `error`](WebAssembly/component-model#405) types. See bytecodealliance/rfcs#38 for more context. Signed-off-by: Joel Dice <[email protected]> add wasmparser::WasmFeatures support to wasm-compose Signed-off-by: Joel Dice <[email protected]> fix no-std build in readers.rs Signed-off-by: Joel Dice <[email protected]> rename `error` to `error-context` per latest spec Signed-off-by: Joel Dice <[email protected]> rename `error` to `error-context` per latest spec (part 2) Also, parse string encoding and realloc from encoded `error-context.new` and `error-context.debug-string` names. Signed-off-by: Joel Dice <[email protected]> add `wast` support for parsing async canon opts And add tests/local/component-model-async/lift-async.wast for round-trip testing of async lifts (more to come). Signed-off-by: Joel Dice <[email protected]> more wast async support and more tests This also fixes a bug in `wasmprinter` keeping track of core functions. Signed-off-by: Joel Dice <[email protected]> more wast async support; add async tests; fix bugs Signed-off-by: Joel Dice <[email protected]> more component-model-async tests and fixes Signed-off-by: Joel Dice <[email protected]> add `wit-parser` tests for streams, futures, and error-contexts Signed-off-by: Joel Dice <[email protected]> add first `wit-component` async test This required adding a new `wit_parser::decoding::decode_reader_with_features` function for passing `WasmFeatures` to `wasmparser::Validator::new_with_features`. Signed-off-by: Joel Dice <[email protected]> add more async tests Signed-off-by: Joel Dice <[email protected]> add `async-builtins` test for `wit-component` Signed-off-by: Joel Dice <[email protected]> add `async-streams-and-futures` test to `wit-component` Signed-off-by: Joel Dice <[email protected]> fix stream/future type handling for exported interfaces Signed-off-by: Joel Dice <[email protected]> support callback-less (AKA stackful) async lifts Signed-off-by: Joel Dice <[email protected]> address review feedback Signed-off-by: Joel Dice <[email protected]>

* Add support for async ABI, futures, streams, and errors This adds support for encoding and parsing components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) and associated canonical options and functions, along with the [`stream`, `future`, and `error`](WebAssembly/component-model#405) types. See bytecodealliance/rfcs#38 for more context. Signed-off-by: Joel Dice <[email protected]> add wasmparser::WasmFeatures support to wasm-compose Signed-off-by: Joel Dice <[email protected]> fix no-std build in readers.rs Signed-off-by: Joel Dice <[email protected]> rename `error` to `error-context` per latest spec Signed-off-by: Joel Dice <[email protected]> rename `error` to `error-context` per latest spec (part 2) Also, parse string encoding and realloc from encoded `error-context.new` and `error-context.debug-string` names. Signed-off-by: Joel Dice <[email protected]> add `wast` support for parsing async canon opts And add tests/local/component-model-async/lift-async.wast for round-trip testing of async lifts (more to come). Signed-off-by: Joel Dice <[email protected]> more wast async support and more tests This also fixes a bug in `wasmprinter` keeping track of core functions. Signed-off-by: Joel Dice <[email protected]> more wast async support; add async tests; fix bugs Signed-off-by: Joel Dice <[email protected]> more component-model-async tests and fixes Signed-off-by: Joel Dice <[email protected]> add `wit-parser` tests for streams, futures, and error-contexts Signed-off-by: Joel Dice <[email protected]> add first `wit-component` async test This required adding a new `wit_parser::decoding::decode_reader_with_features` function for passing `WasmFeatures` to `wasmparser::Validator::new_with_features`. Signed-off-by: Joel Dice <[email protected]> add more async tests Signed-off-by: Joel Dice <[email protected]> add `async-builtins` test for `wit-component` Signed-off-by: Joel Dice <[email protected]> add `async-streams-and-futures` test to `wit-component` Signed-off-by: Joel Dice <[email protected]> fix stream/future type handling for exported interfaces Signed-off-by: Joel Dice <[email protected]> support callback-less (AKA stackful) async lifts Signed-off-by: Joel Dice <[email protected]> address review feedback Signed-off-by: Joel Dice <[email protected]> * address more review feedback Signed-off-by: Joel Dice <[email protected]> * fix test build regression Signed-off-by: Joel Dice <[email protected]> * fix more test regressions Signed-off-by: Joel Dice <[email protected]> * remove need for `Default` impls in `wast::component::func` Signed-off-by: Joel Dice <[email protected]> * fix another test regression Signed-off-by: Joel Dice <[email protected]> * minor code simplification Signed-off-by: Joel Dice <[email protected]> * require wasmparser `features` feature in wasm-compose This fixes `cargo check --no-default-features --feature compose`. Signed-off-by: Joel Dice <[email protected]> * require wasmparser `features` feature in wit-parser Signed-off-by: Joel Dice <[email protected]> * simplify `r#async` parsing code Signed-off-by: Joel Dice <[email protected]> * add a bunch of comments to wit-component async code Signed-off-by: Joel Dice <[email protected]> * allow importing the same function sync and async Signed-off-by: Joel Dice <[email protected]> --------- Signed-off-by: Joel Dice <[email protected]>

…ance#1895) * Add support for async ABI, futures, streams, and errors This adds support for encoding and parsing components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) and associated canonical options and functions, along with the [`stream`, `future`, and `error`](WebAssembly/component-model#405) types. See bytecodealliance/rfcs#38 for more context. Signed-off-by: Joel Dice <[email protected]> add wasmparser::WasmFeatures support to wasm-compose Signed-off-by: Joel Dice <[email protected]> fix no-std build in readers.rs Signed-off-by: Joel Dice <[email protected]> rename `error` to `error-context` per latest spec Signed-off-by: Joel Dice <[email protected]> rename `error` to `error-context` per latest spec (part 2) Also, parse string encoding and realloc from encoded `error-context.new` and `error-context.debug-string` names. Signed-off-by: Joel Dice <[email protected]> add `wast` support for parsing async canon opts And add tests/local/component-model-async/lift-async.wast for round-trip testing of async lifts (more to come). Signed-off-by: Joel Dice <[email protected]> more wast async support and more tests This also fixes a bug in `wasmprinter` keeping track of core functions. Signed-off-by: Joel Dice <[email protected]> more wast async support; add async tests; fix bugs Signed-off-by: Joel Dice <[email protected]> more component-model-async tests and fixes Signed-off-by: Joel Dice <[email protected]> add `wit-parser` tests for streams, futures, and error-contexts Signed-off-by: Joel Dice <[email protected]> add first `wit-component` async test This required adding a new `wit_parser::decoding::decode_reader_with_features` function for passing `WasmFeatures` to `wasmparser::Validator::new_with_features`. Signed-off-by: Joel Dice <[email protected]> add more async tests Signed-off-by: Joel Dice <[email protected]> add `async-builtins` test for `wit-component` Signed-off-by: Joel Dice <[email protected]> add `async-streams-and-futures` test to `wit-component` Signed-off-by: Joel Dice <[email protected]> fix stream/future type handling for exported interfaces Signed-off-by: Joel Dice <[email protected]> support callback-less (AKA stackful) async lifts Signed-off-by: Joel Dice <[email protected]> address review feedback Signed-off-by: Joel Dice <[email protected]> * address more review feedback Signed-off-by: Joel Dice <[email protected]> * fix test build regression Signed-off-by: Joel Dice <[email protected]> * fix more test regressions Signed-off-by: Joel Dice <[email protected]> * remove need for `Default` impls in `wast::component::func` Signed-off-by: Joel Dice <[email protected]> * fix another test regression Signed-off-by: Joel Dice <[email protected]> * minor code simplification Signed-off-by: Joel Dice <[email protected]> * require wasmparser `features` feature in wasm-compose This fixes `cargo check --no-default-features --feature compose`. Signed-off-by: Joel Dice <[email protected]> * require wasmparser `features` feature in wit-parser Signed-off-by: Joel Dice <[email protected]> * simplify `r#async` parsing code Signed-off-by: Joel Dice <[email protected]> * add a bunch of comments to wit-component async code Signed-off-by: Joel Dice <[email protected]> * allow importing the same function sync and async Signed-off-by: Joel Dice <[email protected]> --------- Signed-off-by: Joel Dice <[email protected]>

This adds support for loading, compiling, linking, and running components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) along with the [`stream`, `future`, and `error-context`](WebAssembly/component-model#405) types. It also adds support for generating host bindings such that multiple host functions can be run concurrently with guest tasks -- without monopolizing the `Store`. See the [implementation RFC](bytecodealliance/rfcs#38) for details, as well as [this repo](https://github.com/dicej/component-async-demo) containing end-to-end smoke tests. Signed-off-by: Joel Dice <[email protected]> fix clippy warnings and bench/fuzzing errors Signed-off-by: Joel Dice <[email protected]> revert atomic.wit whitespace change Signed-off-by: Joel Dice <[email protected]> fix build when component-model disabled Signed-off-by: Joel Dice <[email protected]> bless component-macro expected output Signed-off-by: Joel Dice <[email protected]> fix no-std build error Signed-off-by: Joel Dice <[email protected]> fix build with --no-default-features --features runtime,component-model Signed-off-by: Joel Dice <[email protected]> partly fix no-std build It's still broken due to the use of `std::collections::HashMap` in crates/wasmtime/src/runtime/vm/component.rs. I'll address that as part of the work to avoid exposing global task/future/stream/error-context handles to guests. Signed-off-by: Joel Dice <[email protected]> maintain per-instance tables for futures, streams, and error-contexts Signed-off-by: Joel Dice <[email protected]> refactor task/stream/future handle lifting/lowering This addresses a couple of issues: - Previously, we were passing task/stream/future/error-context reps directly to instances while keeping track of which instance had access to which rep. That worked fine in that there was no way to forge access to inaccessible reps, but it leaked information about what other instances were doing. Now we maintain per-instance waitable and error-context tables which map the reps to and from the handles which the instance sees. - The `no_std` build was broken due to use of `HashMap` in `runtime::vm::component`, which is now fixed. Note that we use one single table per instance for all tasks, streams, and futures. This is partly necessary because, when async events are delivered to the guest, it wouldn't have enough context to know which stream or future we're talking about if each unique stream and future type had its own table. So at minimum, we need to use the same table for all streams (regardless of payload type), and likewise for futures. Also, per WebAssembly/component-model#395 (comment), the plan is to move towards a shared table for all resource types as well, so this moves us in that direction. Signed-off-by: Joel Dice <[email protected]> fix wave breakage due to new stream/future/error-context types Signed-off-by: Joel Dice <[email protected]> switch wasm-tools to v1.220.0-based branch Signed-off-by: Joel Dice <[email protected]> check `task.return` type at runtime We can't statically verify a given call to `task.return` corresponds to the expected core signature appropriate for the currently running task, so we must do so at runtime. In order to make that check efficient, we intern the types. My initial plan was to use `ModuleInternedTypeIndex` and/or `VMSharedTypeIndex` for interning, but that got hairy with WasmGC considerations, so instead I added new fields to `ComponentTypes` and `ComponentTypesBuilder`. Signed-off-by: Joel Dice <[email protected]> add `TypedFunc::call_concurrent` and refine stream/future APIs This implements what I proposed in https://github.com/dicej/rfcs/blob/component-async/accepted/component-model-async.md#wasmtime. Specifically, it adds: - A new `Promise` type, useful for working with concurrent operations that require access to a `Store` to make progress. - A new `PromisesUnordered` type for `await`ing multiple promises concurrently -`TypedFunc::call_concurrent` (which returns a `Promise`), allowing multiple host->guest calls to run concurrently on the same instance. - Updated `{Stream|Future}{Writer|Reader}` APIs which use `Promise` The upshot is that the embedder can now ergonomically manage arbitrary numbers of concurrent operations. Previously, this was a lot more difficult to do without accidentally starving some of the operations due to another one monopolizing the `Store`. Finally, this includes various refactorings and fixes for bugs exposed by the newer, more versatile APIs. Signed-off-by: Joel Dice <[email protected]> clean up verbosity in component/func.rs Signed-off-by: Joel Dice <[email protected]> snapshot Signed-off-by: Joel Dice <[email protected]> implement stream/future read/write cancellation This required a somewhat viral addition of `Send` and `Sync` bounds for async host function closure types, unfortunately. Signed-off-by: Joel Dice <[email protected]> add `Func::call_concurrent` and `LinkerInstance::func_new_concurrent` Signed-off-by: Joel Dice <[email protected]> dynamic API support for streams/futures/error-contexts Signed-off-by: Joel Dice <[email protected]> support callback-less (AKA stackful) async lifts Signed-off-by: Joel Dice <[email protected]> fix `call_host` regression Signed-off-by: Joel Dice <[email protected]> add component model async end-to-end tests I've ported these over from https://github.com/dicej/component-async-demo Signed-off-by: Joel Dice <[email protected]> fix test regressions and clippy warnings Signed-off-by: Joel Dice <[email protected]> satisfy clippy Signed-off-by: Joel Dice <[email protected]>

This adds support for loading, compiling, linking, and running components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) along with the [`stream`, `future`, and `error-context`](WebAssembly/component-model#405) types. It also adds support for generating host bindings such that multiple host functions can be run concurrently with guest tasks -- without monopolizing the `Store`. See the [implementation RFC](bytecodealliance/rfcs#38) for details, as well as [this repo](https://github.com/dicej/component-async-demo) containing end-to-end smoke tests. Signed-off-by: Joel Dice <[email protected]> fix clippy warnings and bench/fuzzing errors Signed-off-by: Joel Dice <[email protected]> revert atomic.wit whitespace change Signed-off-by: Joel Dice <[email protected]> fix build when component-model disabled Signed-off-by: Joel Dice <[email protected]> bless component-macro expected output Signed-off-by: Joel Dice <[email protected]> fix no-std build error Signed-off-by: Joel Dice <[email protected]> fix build with --no-default-features --features runtime,component-model Signed-off-by: Joel Dice <[email protected]> partly fix no-std build It's still broken due to the use of `std::collections::HashMap` in crates/wasmtime/src/runtime/vm/component.rs. I'll address that as part of the work to avoid exposing global task/future/stream/error-context handles to guests. Signed-off-by: Joel Dice <[email protected]> maintain per-instance tables for futures, streams, and error-contexts Signed-off-by: Joel Dice <[email protected]> refactor task/stream/future handle lifting/lowering This addresses a couple of issues: - Previously, we were passing task/stream/future/error-context reps directly to instances while keeping track of which instance had access to which rep. That worked fine in that there was no way to forge access to inaccessible reps, but it leaked information about what other instances were doing. Now we maintain per-instance waitable and error-context tables which map the reps to and from the handles which the instance sees. - The `no_std` build was broken due to use of `HashMap` in `runtime::vm::component`, which is now fixed. Note that we use one single table per instance for all tasks, streams, and futures. This is partly necessary because, when async events are delivered to the guest, it wouldn't have enough context to know which stream or future we're talking about if each unique stream and future type had its own table. So at minimum, we need to use the same table for all streams (regardless of payload type), and likewise for futures. Also, per WebAssembly/component-model#395 (comment), the plan is to move towards a shared table for all resource types as well, so this moves us in that direction. Signed-off-by: Joel Dice <[email protected]> fix wave breakage due to new stream/future/error-context types Signed-off-by: Joel Dice <[email protected]> switch wasm-tools to v1.220.0-based branch Signed-off-by: Joel Dice <[email protected]> check `task.return` type at runtime We can't statically verify a given call to `task.return` corresponds to the expected core signature appropriate for the currently running task, so we must do so at runtime. In order to make that check efficient, we intern the types. My initial plan was to use `ModuleInternedTypeIndex` and/or `VMSharedTypeIndex` for interning, but that got hairy with WasmGC considerations, so instead I added new fields to `ComponentTypes` and `ComponentTypesBuilder`. Signed-off-by: Joel Dice <[email protected]> add `TypedFunc::call_concurrent` and refine stream/future APIs This implements what I proposed in https://github.com/dicej/rfcs/blob/component-async/accepted/component-model-async.md#wasmtime. Specifically, it adds: - A new `Promise` type, useful for working with concurrent operations that require access to a `Store` to make progress. - A new `PromisesUnordered` type for `await`ing multiple promises concurrently -`TypedFunc::call_concurrent` (which returns a `Promise`), allowing multiple host->guest calls to run concurrently on the same instance. - Updated `{Stream|Future}{Writer|Reader}` APIs which use `Promise` The upshot is that the embedder can now ergonomically manage arbitrary numbers of concurrent operations. Previously, this was a lot more difficult to do without accidentally starving some of the operations due to another one monopolizing the `Store`. Finally, this includes various refactorings and fixes for bugs exposed by the newer, more versatile APIs. Signed-off-by: Joel Dice <[email protected]> clean up verbosity in component/func.rs Signed-off-by: Joel Dice <[email protected]> snapshot Signed-off-by: Joel Dice <[email protected]> implement stream/future read/write cancellation This required a somewhat viral addition of `Send` and `Sync` bounds for async host function closure types, unfortunately. Signed-off-by: Joel Dice <[email protected]> add `Func::call_concurrent` and `LinkerInstance::func_new_concurrent` Signed-off-by: Joel Dice <[email protected]> dynamic API support for streams/futures/error-contexts Signed-off-by: Joel Dice <[email protected]> support callback-less (AKA stackful) async lifts Signed-off-by: Joel Dice <[email protected]> fix `call_host` regression Signed-off-by: Joel Dice <[email protected]> add component model async end-to-end tests I've ported these over from https://github.com/dicej/component-async-demo Signed-off-by: Joel Dice <[email protected]> fix test regressions and clippy warnings Signed-off-by: Joel Dice <[email protected]> satisfy clippy Signed-off-by: Joel Dice <[email protected]> fix async tests when `component-model-async` enabled Enabling this feature for all tests revealed various missing pieces in the new `concurrent.rs` fiber mechanism, which I've addressed. This adds a bunch of ugly `#[cfg(feature = "component-model-async")]` guards, but those will all go away once I unify the two async fiber implementations. Signed-off-by: Joel Dice <[email protected]>

This adds support for loading, compiling, linking, and running components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) along with the [`stream`, `future`, and `error-context`](WebAssembly/component-model#405) types. It also adds support for generating host bindings such that multiple host functions can be run concurrently with guest tasks -- without monopolizing the `Store`. See the [implementation RFC](bytecodealliance/rfcs#38) for details, as well as [this repo](https://github.com/dicej/component-async-demo) containing end-to-end smoke tests. Signed-off-by: Joel Dice <[email protected]> fix clippy warnings and bench/fuzzing errors Signed-off-by: Joel Dice <[email protected]> revert atomic.wit whitespace change Signed-off-by: Joel Dice <[email protected]> fix build when component-model disabled Signed-off-by: Joel Dice <[email protected]> bless component-macro expected output Signed-off-by: Joel Dice <[email protected]> fix no-std build error Signed-off-by: Joel Dice <[email protected]> fix build with --no-default-features --features runtime,component-model Signed-off-by: Joel Dice <[email protected]> partly fix no-std build It's still broken due to the use of `std::collections::HashMap` in crates/wasmtime/src/runtime/vm/component.rs. I'll address that as part of the work to avoid exposing global task/future/stream/error-context handles to guests. Signed-off-by: Joel Dice <[email protected]> maintain per-instance tables for futures, streams, and error-contexts Signed-off-by: Joel Dice <[email protected]> refactor task/stream/future handle lifting/lowering This addresses a couple of issues: - Previously, we were passing task/stream/future/error-context reps directly to instances while keeping track of which instance had access to which rep. That worked fine in that there was no way to forge access to inaccessible reps, but it leaked information about what other instances were doing. Now we maintain per-instance waitable and error-context tables which map the reps to and from the handles which the instance sees. - The `no_std` build was broken due to use of `HashMap` in `runtime::vm::component`, which is now fixed. Note that we use one single table per instance for all tasks, streams, and futures. This is partly necessary because, when async events are delivered to the guest, it wouldn't have enough context to know which stream or future we're talking about if each unique stream and future type had its own table. So at minimum, we need to use the same table for all streams (regardless of payload type), and likewise for futures. Also, per WebAssembly/component-model#395 (comment), the plan is to move towards a shared table for all resource types as well, so this moves us in that direction. Signed-off-by: Joel Dice <[email protected]> fix wave breakage due to new stream/future/error-context types Signed-off-by: Joel Dice <[email protected]> switch wasm-tools to v1.220.0-based branch Signed-off-by: Joel Dice <[email protected]> check `task.return` type at runtime We can't statically verify a given call to `task.return` corresponds to the expected core signature appropriate for the currently running task, so we must do so at runtime. In order to make that check efficient, we intern the types. My initial plan was to use `ModuleInternedTypeIndex` and/or `VMSharedTypeIndex` for interning, but that got hairy with WasmGC considerations, so instead I added new fields to `ComponentTypes` and `ComponentTypesBuilder`. Signed-off-by: Joel Dice <[email protected]> add `TypedFunc::call_concurrent` and refine stream/future APIs This implements what I proposed in https://github.com/dicej/rfcs/blob/component-async/accepted/component-model-async.md#wasmtime. Specifically, it adds: - A new `Promise` type, useful for working with concurrent operations that require access to a `Store` to make progress. - A new `PromisesUnordered` type for `await`ing multiple promises concurrently -`TypedFunc::call_concurrent` (which returns a `Promise`), allowing multiple host->guest calls to run concurrently on the same instance. - Updated `{Stream|Future}{Writer|Reader}` APIs which use `Promise` The upshot is that the embedder can now ergonomically manage arbitrary numbers of concurrent operations. Previously, this was a lot more difficult to do without accidentally starving some of the operations due to another one monopolizing the `Store`. Finally, this includes various refactorings and fixes for bugs exposed by the newer, more versatile APIs. Signed-off-by: Joel Dice <[email protected]> clean up verbosity in component/func.rs Signed-off-by: Joel Dice <[email protected]> snapshot Signed-off-by: Joel Dice <[email protected]> implement stream/future read/write cancellation This required a somewhat viral addition of `Send` and `Sync` bounds for async host function closure types, unfortunately. Signed-off-by: Joel Dice <[email protected]> add `Func::call_concurrent` and `LinkerInstance::func_new_concurrent` Signed-off-by: Joel Dice <[email protected]> dynamic API support for streams/futures/error-contexts Signed-off-by: Joel Dice <[email protected]> support callback-less (AKA stackful) async lifts Signed-off-by: Joel Dice <[email protected]> fix `call_host` regression Signed-off-by: Joel Dice <[email protected]> add component model async end-to-end tests I've ported these over from https://github.com/dicej/component-async-demo Signed-off-by: Joel Dice <[email protected]> fix test regressions and clippy warnings Signed-off-by: Joel Dice <[email protected]> satisfy clippy Signed-off-by: Joel Dice <[email protected]> fix async tests when `component-model-async` enabled Enabling this feature for all tests revealed various missing pieces in the new `concurrent.rs` fiber mechanism, which I've addressed. This adds a bunch of ugly `#[cfg(feature = "component-model-async")]` guards, but those will all go away once I unify the two async fiber implementations. Signed-off-by: Joel Dice <[email protected]> add and modify tests to cover concurrent APIs Primarily, this tests and implements cases where parameters and/or results must be passed via linear memory instead of the stack. Signed-off-by: Joel Dice <[email protected]> `concurrent_{imports|exports}` component macro codegen tests This enables codegen testing of the `concurrent_imports` and `concurrent_exports` options to `wasmtime::component::bindgen` and also fixes code generation for world-level function and resource exports that use the concurrent call style. Signed-off-by: Joel Dice <[email protected]> `concurrent_{imports|exports}` component macro expanded tests This enables testing of the `concurrent_imports` and `concurrent_exports` options in `crates/component-macro/tests/expanded.rs`. Signed-off-by: Joel Dice <[email protected]> add tests/misc_testsuite/component-model-async/*.wast These only test instantiation of components which use various async options and built-ins so far. Next, I'll happy and sad path tests which actually execute code. Signed-off-by: Joel Dice <[email protected]> appease clippy Signed-off-by: Joel Dice <[email protected]> add tests/misc_testsuite/component-model-async/fused.wast Signed-off-by: Joel Dice <[email protected]> add non-panicking bounds checks where appropriate Signed-off-by: Joel Dice <[email protected]>

This adds support for loading, compiling, linking, and running components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) along with the [`stream`, `future`, and `error-context`](WebAssembly/component-model#405) types. It also adds support for generating host bindings such that multiple host functions can be run concurrently with guest tasks -- without monopolizing the `Store`. See the [implementation RFC](bytecodealliance/rfcs#38) for details, as well as [this repo](https://github.com/dicej/component-async-demo) containing end-to-end smoke tests. Signed-off-by: Joel Dice <[email protected]> fix clippy warnings and bench/fuzzing errors Signed-off-by: Joel Dice <[email protected]> revert atomic.wit whitespace change Signed-off-by: Joel Dice <[email protected]> fix build when component-model disabled Signed-off-by: Joel Dice <[email protected]> bless component-macro expected output Signed-off-by: Joel Dice <[email protected]> fix no-std build error Signed-off-by: Joel Dice <[email protected]> fix build with --no-default-features --features runtime,component-model Signed-off-by: Joel Dice <[email protected]> partly fix no-std build It's still broken due to the use of `std::collections::HashMap` in crates/wasmtime/src/runtime/vm/component.rs. I'll address that as part of the work to avoid exposing global task/future/stream/error-context handles to guests. Signed-off-by: Joel Dice <[email protected]> maintain per-instance tables for futures, streams, and error-contexts Signed-off-by: Joel Dice <[email protected]> refactor task/stream/future handle lifting/lowering This addresses a couple of issues: - Previously, we were passing task/stream/future/error-context reps directly to instances while keeping track of which instance had access to which rep. That worked fine in that there was no way to forge access to inaccessible reps, but it leaked information about what other instances were doing. Now we maintain per-instance waitable and error-context tables which map the reps to and from the handles which the instance sees. - The `no_std` build was broken due to use of `HashMap` in `runtime::vm::component`, which is now fixed. Note that we use one single table per instance for all tasks, streams, and futures. This is partly necessary because, when async events are delivered to the guest, it wouldn't have enough context to know which stream or future we're talking about if each unique stream and future type had its own table. So at minimum, we need to use the same table for all streams (regardless of payload type), and likewise for futures. Also, per WebAssembly/component-model#395 (comment), the plan is to move towards a shared table for all resource types as well, so this moves us in that direction. Signed-off-by: Joel Dice <[email protected]> fix wave breakage due to new stream/future/error-context types Signed-off-by: Joel Dice <[email protected]> switch wasm-tools to v1.220.0-based branch Signed-off-by: Joel Dice <[email protected]> check `task.return` type at runtime We can't statically verify a given call to `task.return` corresponds to the expected core signature appropriate for the currently running task, so we must do so at runtime. In order to make that check efficient, we intern the types. My initial plan was to use `ModuleInternedTypeIndex` and/or `VMSharedTypeIndex` for interning, but that got hairy with WasmGC considerations, so instead I added new fields to `ComponentTypes` and `ComponentTypesBuilder`. Signed-off-by: Joel Dice <[email protected]> add `TypedFunc::call_concurrent` and refine stream/future APIs This implements what I proposed in https://github.com/dicej/rfcs/blob/component-async/accepted/component-model-async.md#wasmtime. Specifically, it adds: - A new `Promise` type, useful for working with concurrent operations that require access to a `Store` to make progress. - A new `PromisesUnordered` type for `await`ing multiple promises concurrently -`TypedFunc::call_concurrent` (which returns a `Promise`), allowing multiple host->guest calls to run concurrently on the same instance. - Updated `{Stream|Future}{Writer|Reader}` APIs which use `Promise` The upshot is that the embedder can now ergonomically manage arbitrary numbers of concurrent operations. Previously, this was a lot more difficult to do without accidentally starving some of the operations due to another one monopolizing the `Store`. Finally, this includes various refactorings and fixes for bugs exposed by the newer, more versatile APIs. Signed-off-by: Joel Dice <[email protected]> clean up verbosity in component/func.rs Signed-off-by: Joel Dice <[email protected]> snapshot Signed-off-by: Joel Dice <[email protected]> implement stream/future read/write cancellation This required a somewhat viral addition of `Send` and `Sync` bounds for async host function closure types, unfortunately. Signed-off-by: Joel Dice <[email protected]> add `Func::call_concurrent` and `LinkerInstance::func_new_concurrent` Signed-off-by: Joel Dice <[email protected]> dynamic API support for streams/futures/error-contexts Signed-off-by: Joel Dice <[email protected]> support callback-less (AKA stackful) async lifts Signed-off-by: Joel Dice <[email protected]> fix `call_host` regression Signed-off-by: Joel Dice <[email protected]> add component model async end-to-end tests I've ported these over from https://github.com/dicej/component-async-demo Signed-off-by: Joel Dice <[email protected]> fix test regressions and clippy warnings Signed-off-by: Joel Dice <[email protected]> satisfy clippy Signed-off-by: Joel Dice <[email protected]> fix async tests when `component-model-async` enabled Enabling this feature for all tests revealed various missing pieces in the new `concurrent.rs` fiber mechanism, which I've addressed. This adds a bunch of ugly `#[cfg(feature = "component-model-async")]` guards, but those will all go away once I unify the two async fiber implementations. Signed-off-by: Joel Dice <[email protected]> add and modify tests to cover concurrent APIs Primarily, this tests and implements cases where parameters and/or results must be passed via linear memory instead of the stack. Signed-off-by: Joel Dice <[email protected]> `concurrent_{imports|exports}` component macro codegen tests This enables codegen testing of the `concurrent_imports` and `concurrent_exports` options to `wasmtime::component::bindgen` and also fixes code generation for world-level function and resource exports that use the concurrent call style. Signed-off-by: Joel Dice <[email protected]> `concurrent_{imports|exports}` component macro expanded tests This enables testing of the `concurrent_imports` and `concurrent_exports` options in `crates/component-macro/tests/expanded.rs`. Signed-off-by: Joel Dice <[email protected]> add tests/misc_testsuite/component-model-async/*.wast These only test instantiation of components which use various async options and built-ins so far. Next, I'll happy and sad path tests which actually execute code. Signed-off-by: Joel Dice <[email protected]> appease clippy Signed-off-by: Joel Dice <[email protected]> add tests/misc_testsuite/component-model-async/fused.wast Signed-off-by: Joel Dice <[email protected]> add non-panicking bounds checks where appropriate Signed-off-by: Joel Dice <[email protected]> remove post-return bits from async result lift code ...at least until we've determined whether post-return options even make sense for async-lifted exports. Signed-off-by: Joel Dice <[email protected]> fix component-model-async/fused.wast test failure Signed-off-by: Joel Dice <[email protected]> use `enum` types to represent status and event codes Signed-off-by: Joel Dice <[email protected]> fix component-model-async/fused.wast test failure (2nd try) Signed-off-by: Joel Dice <[email protected]> use `gc_types = true` in component-model-async/fused.wast We use `Instruction::RefFunc` when generating adapters for async lifts and/or lowers, which Winch doesn't understand, and apparently `gc_types = true` is what tells the test infra not to use Winch. Signed-off-by: Joel Dice <[email protected]> trap if async function finishes without calling `task.return` Signed-off-by: Joel Dice <[email protected]> update wit-bindgen and fix rebase damage Signed-off-by: Joel Dice <[email protected]> call post-return function if any for async->sync fused calls Signed-off-by: Joel Dice <[email protected]>

This adds support for loading, compiling, linking, and running components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) along with the [`stream`, `future`, and `error-context`](WebAssembly/component-model#405) types. It also adds support for generating host bindings such that multiple host functions can be run concurrently with guest tasks -- without monopolizing the `Store`. See the [implementation RFC](bytecodealliance/rfcs#38) for details, as well as [this repo](https://github.com/dicej/component-async-demo) containing end-to-end smoke tests. Signed-off-by: Joel Dice <[email protected]> fix clippy warnings and bench/fuzzing errors Signed-off-by: Joel Dice <[email protected]> revert atomic.wit whitespace change Signed-off-by: Joel Dice <[email protected]> fix build when component-model disabled Signed-off-by: Joel Dice <[email protected]> bless component-macro expected output Signed-off-by: Joel Dice <[email protected]> fix no-std build error Signed-off-by: Joel Dice <[email protected]> fix build with --no-default-features --features runtime,component-model Signed-off-by: Joel Dice <[email protected]> partly fix no-std build It's still broken due to the use of `std::collections::HashMap` in crates/wasmtime/src/runtime/vm/component.rs. I'll address that as part of the work to avoid exposing global task/future/stream/error-context handles to guests. Signed-off-by: Joel Dice <[email protected]> maintain per-instance tables for futures, streams, and error-contexts Signed-off-by: Joel Dice <[email protected]> refactor task/stream/future handle lifting/lowering This addresses a couple of issues: - Previously, we were passing task/stream/future/error-context reps directly to instances while keeping track of which instance had access to which rep. That worked fine in that there was no way to forge access to inaccessible reps, but it leaked information about what other instances were doing. Now we maintain per-instance waitable and error-context tables which map the reps to and from the handles which the instance sees. - The `no_std` build was broken due to use of `HashMap` in `runtime::vm::component`, which is now fixed. Note that we use one single table per instance for all tasks, streams, and futures. This is partly necessary because, when async events are delivered to the guest, it wouldn't have enough context to know which stream or future we're talking about if each unique stream and future type had its own table. So at minimum, we need to use the same table for all streams (regardless of payload type), and likewise for futures. Also, per WebAssembly/component-model#395 (comment), the plan is to move towards a shared table for all resource types as well, so this moves us in that direction. Signed-off-by: Joel Dice <[email protected]> fix wave breakage due to new stream/future/error-context types Signed-off-by: Joel Dice <[email protected]> switch wasm-tools to v1.220.0-based branch Signed-off-by: Joel Dice <[email protected]> check `task.return` type at runtime We can't statically verify a given call to `task.return` corresponds to the expected core signature appropriate for the currently running task, so we must do so at runtime. In order to make that check efficient, we intern the types. My initial plan was to use `ModuleInternedTypeIndex` and/or `VMSharedTypeIndex` for interning, but that got hairy with WasmGC considerations, so instead I added new fields to `ComponentTypes` and `ComponentTypesBuilder`. Signed-off-by: Joel Dice <[email protected]> add `TypedFunc::call_concurrent` and refine stream/future APIs This implements what I proposed in https://github.com/dicej/rfcs/blob/component-async/accepted/component-model-async.md#wasmtime. Specifically, it adds: - A new `Promise` type, useful for working with concurrent operations that require access to a `Store` to make progress. - A new `PromisesUnordered` type for `await`ing multiple promises concurrently -`TypedFunc::call_concurrent` (which returns a `Promise`), allowing multiple host->guest calls to run concurrently on the same instance. - Updated `{Stream|Future}{Writer|Reader}` APIs which use `Promise` The upshot is that the embedder can now ergonomically manage arbitrary numbers of concurrent operations. Previously, this was a lot more difficult to do without accidentally starving some of the operations due to another one monopolizing the `Store`. Finally, this includes various refactorings and fixes for bugs exposed by the newer, more versatile APIs. Signed-off-by: Joel Dice <[email protected]> clean up verbosity in component/func.rs Signed-off-by: Joel Dice <[email protected]> snapshot Signed-off-by: Joel Dice <[email protected]> implement stream/future read/write cancellation This required a somewhat viral addition of `Send` and `Sync` bounds for async host function closure types, unfortunately. Signed-off-by: Joel Dice <[email protected]> add `Func::call_concurrent` and `LinkerInstance::func_new_concurrent` Signed-off-by: Joel Dice <[email protected]> dynamic API support for streams/futures/error-contexts Signed-off-by: Joel Dice <[email protected]> support callback-less (AKA stackful) async lifts Signed-off-by: Joel Dice <[email protected]> fix `call_host` regression Signed-off-by: Joel Dice <[email protected]> add component model async end-to-end tests I've ported these over from https://github.com/dicej/component-async-demo Signed-off-by: Joel Dice <[email protected]> fix test regressions and clippy warnings Signed-off-by: Joel Dice <[email protected]> satisfy clippy Signed-off-by: Joel Dice <[email protected]> fix async tests when `component-model-async` enabled Enabling this feature for all tests revealed various missing pieces in the new `concurrent.rs` fiber mechanism, which I've addressed. This adds a bunch of ugly `#[cfg(feature = "component-model-async")]` guards, but those will all go away once I unify the two async fiber implementations. Signed-off-by: Joel Dice <[email protected]> add and modify tests to cover concurrent APIs Primarily, this tests and implements cases where parameters and/or results must be passed via linear memory instead of the stack. Signed-off-by: Joel Dice <[email protected]> `concurrent_{imports|exports}` component macro codegen tests This enables codegen testing of the `concurrent_imports` and `concurrent_exports` options to `wasmtime::component::bindgen` and also fixes code generation for world-level function and resource exports that use the concurrent call style. Signed-off-by: Joel Dice <[email protected]> `concurrent_{imports|exports}` component macro expanded tests This enables testing of the `concurrent_imports` and `concurrent_exports` options in `crates/component-macro/tests/expanded.rs`. Signed-off-by: Joel Dice <[email protected]> add tests/misc_testsuite/component-model-async/*.wast These only test instantiation of components which use various async options and built-ins so far. Next, I'll happy and sad path tests which actually execute code. Signed-off-by: Joel Dice <[email protected]> appease clippy Signed-off-by: Joel Dice <[email protected]> add tests/misc_testsuite/component-model-async/fused.wast Signed-off-by: Joel Dice <[email protected]> add non-panicking bounds checks where appropriate Signed-off-by: Joel Dice <[email protected]> remove post-return bits from async result lift code ...at least until we've determined whether post-return options even make sense for async-lifted exports. Signed-off-by: Joel Dice <[email protected]> fix component-model-async/fused.wast test failure Signed-off-by: Joel Dice <[email protected]> use `enum` types to represent status and event codes Signed-off-by: Joel Dice <[email protected]> fix component-model-async/fused.wast test failure (2nd try) Signed-off-by: Joel Dice <[email protected]> use `gc_types = true` in component-model-async/fused.wast We use `Instruction::RefFunc` when generating adapters for async lifts and/or lowers, which Winch doesn't understand, and apparently `gc_types = true` is what tells the test infra not to use Winch. Signed-off-by: Joel Dice <[email protected]> trap if async function finishes without calling `task.return` Signed-off-by: Joel Dice <[email protected]> update wit-bindgen and fix rebase damage Signed-off-by: Joel Dice <[email protected]> call post-return function if any for async->sync fused calls Signed-off-by: Joel Dice <[email protected]> fix non-component-model-async build; appease clippy Signed-off-by: Joel Dice <[email protected]> bless bindgen output whitespace changes Signed-off-by: Joel Dice <[email protected]> enforce resource borrow requirements for async calls Signed-off-by: Joel Dice <[email protected]> update `wit-bindgen` and simplify `async_borrowing_callee` test Signed-off-by: Joel Dice <[email protected]> call `InstanceFlags::set_may_enter` where appropriate There's still more work to do to fully implement (and test) the reentrance rules for concurrent tasks, but this is a start. Signed-off-by: Joel Dice <[email protected]> finish implementing reentrance checks Signed-off-by: Joel Dice <[email protected]>

This adds support for loading, compiling, linking, and running components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) along with the [`stream`, `future`, and `error-context`](WebAssembly/component-model#405) types. It also adds support for generating host bindings such that multiple host functions can be run concurrently with guest tasks -- without monopolizing the `Store`. See the [implementation RFC](bytecodealliance/rfcs#38) for details, as well as [this repo](https://github.com/dicej/component-async-demo) containing end-to-end smoke tests. Signed-off-by: Joel Dice <[email protected]> fix clippy warnings and bench/fuzzing errors Signed-off-by: Joel Dice <[email protected]> revert atomic.wit whitespace change Signed-off-by: Joel Dice <[email protected]> fix build when component-model disabled Signed-off-by: Joel Dice <[email protected]> bless component-macro expected output Signed-off-by: Joel Dice <[email protected]> fix no-std build error Signed-off-by: Joel Dice <[email protected]> fix build with --no-default-features --features runtime,component-model Signed-off-by: Joel Dice <[email protected]> partly fix no-std build It's still broken due to the use of `std::collections::HashMap` in crates/wasmtime/src/runtime/vm/component.rs. I'll address that as part of the work to avoid exposing global task/future/stream/error-context handles to guests. Signed-off-by: Joel Dice <[email protected]> maintain per-instance tables for futures, streams, and error-contexts Signed-off-by: Joel Dice <[email protected]> refactor task/stream/future handle lifting/lowering This addresses a couple of issues: - Previously, we were passing task/stream/future/error-context reps directly to instances while keeping track of which instance had access to which rep. That worked fine in that there was no way to forge access to inaccessible reps, but it leaked information about what other instances were doing. Now we maintain per-instance waitable and error-context tables which map the reps to and from the handles which the instance sees. - The `no_std` build was broken due to use of `HashMap` in `runtime::vm::component`, which is now fixed. Note that we use one single table per instance for all tasks, streams, and futures. This is partly necessary because, when async events are delivered to the guest, it wouldn't have enough context to know which stream or future we're talking about if each unique stream and future type had its own table. So at minimum, we need to use the same table for all streams (regardless of payload type), and likewise for futures. Also, per WebAssembly/component-model#395 (comment), the plan is to move towards a shared table for all resource types as well, so this moves us in that direction. Signed-off-by: Joel Dice <[email protected]> fix wave breakage due to new stream/future/error-context types Signed-off-by: Joel Dice <[email protected]> switch wasm-tools to v1.220.0-based branch Signed-off-by: Joel Dice <[email protected]> check `task.return` type at runtime We can't statically verify a given call to `task.return` corresponds to the expected core signature appropriate for the currently running task, so we must do so at runtime. In order to make that check efficient, we intern the types. My initial plan was to use `ModuleInternedTypeIndex` and/or `VMSharedTypeIndex` for interning, but that got hairy with WasmGC considerations, so instead I added new fields to `ComponentTypes` and `ComponentTypesBuilder`. Signed-off-by: Joel Dice <[email protected]> add `TypedFunc::call_concurrent` and refine stream/future APIs This implements what I proposed in https://github.com/dicej/rfcs/blob/component-async/accepted/component-model-async.md#wasmtime. Specifically, it adds: - A new `Promise` type, useful for working with concurrent operations that require access to a `Store` to make progress. - A new `PromisesUnordered` type for `await`ing multiple promises concurrently -`TypedFunc::call_concurrent` (which returns a `Promise`), allowing multiple host->guest calls to run concurrently on the same instance. - Updated `{Stream|Future}{Writer|Reader}` APIs which use `Promise` The upshot is that the embedder can now ergonomically manage arbitrary numbers of concurrent operations. Previously, this was a lot more difficult to do without accidentally starving some of the operations due to another one monopolizing the `Store`. Finally, this includes various refactorings and fixes for bugs exposed by the newer, more versatile APIs. Signed-off-by: Joel Dice <[email protected]> clean up verbosity in component/func.rs Signed-off-by: Joel Dice <[email protected]> snapshot Signed-off-by: Joel Dice <[email protected]> implement stream/future read/write cancellation This required a somewhat viral addition of `Send` and `Sync` bounds for async host function closure types, unfortunately. Signed-off-by: Joel Dice <[email protected]> add `Func::call_concurrent` and `LinkerInstance::func_new_concurrent` Signed-off-by: Joel Dice <[email protected]> dynamic API support for streams/futures/error-contexts Signed-off-by: Joel Dice <[email protected]> support callback-less (AKA stackful) async lifts Signed-off-by: Joel Dice <[email protected]> fix `call_host` regression Signed-off-by: Joel Dice <[email protected]> add component model async end-to-end tests I've ported these over from https://github.com/dicej/component-async-demo Signed-off-by: Joel Dice <[email protected]> fix test regressions and clippy warnings Signed-off-by: Joel Dice <[email protected]> satisfy clippy Signed-off-by: Joel Dice <[email protected]> fix async tests when `component-model-async` enabled Enabling this feature for all tests revealed various missing pieces in the new `concurrent.rs` fiber mechanism, which I've addressed. This adds a bunch of ugly `#[cfg(feature = "component-model-async")]` guards, but those will all go away once I unify the two async fiber implementations. Signed-off-by: Joel Dice <[email protected]> add and modify tests to cover concurrent APIs Primarily, this tests and implements cases where parameters and/or results must be passed via linear memory instead of the stack. Signed-off-by: Joel Dice <[email protected]> `concurrent_{imports|exports}` component macro codegen tests This enables codegen testing of the `concurrent_imports` and `concurrent_exports` options to `wasmtime::component::bindgen` and also fixes code generation for world-level function and resource exports that use the concurrent call style. Signed-off-by: Joel Dice <[email protected]> `concurrent_{imports|exports}` component macro expanded tests This enables testing of the `concurrent_imports` and `concurrent_exports` options in `crates/component-macro/tests/expanded.rs`. Signed-off-by: Joel Dice <[email protected]> add tests/misc_testsuite/component-model-async/*.wast These only test instantiation of components which use various async options and built-ins so far. Next, I'll happy and sad path tests which actually execute code. Signed-off-by: Joel Dice <[email protected]> appease clippy Signed-off-by: Joel Dice <[email protected]> add tests/misc_testsuite/component-model-async/fused.wast Signed-off-by: Joel Dice <[email protected]> add non-panicking bounds checks where appropriate Signed-off-by: Joel Dice <[email protected]> remove post-return bits from async result lift code ...at least until we've determined whether post-return options even make sense for async-lifted exports. Signed-off-by: Joel Dice <[email protected]> fix component-model-async/fused.wast test failure Signed-off-by: Joel Dice <[email protected]> use `enum` types to represent status and event codes Signed-off-by: Joel Dice <[email protected]> fix component-model-async/fused.wast test failure (2nd try) Signed-off-by: Joel Dice <[email protected]> use `gc_types = true` in component-model-async/fused.wast We use `Instruction::RefFunc` when generating adapters for async lifts and/or lowers, which Winch doesn't understand, and apparently `gc_types = true` is what tells the test infra not to use Winch. Signed-off-by: Joel Dice <[email protected]> trap if async function finishes without calling `task.return` Signed-off-by: Joel Dice <[email protected]> update wit-bindgen and fix rebase damage Signed-off-by: Joel Dice <[email protected]> call post-return function if any for async->sync fused calls Signed-off-by: Joel Dice <[email protected]>

This adds support for loading, compiling, linking, and running components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) along with the [`stream`, `future`, and `error-context`](WebAssembly/component-model#405) types. It also adds support for generating host bindings such that multiple host functions can be run concurrently with guest tasks -- without monopolizing the `Store`. See the [implementation RFC](bytecodealliance/rfcs#38) for details, as well as [this repo](https://github.com/dicej/component-async-demo) containing end-to-end smoke tests. Signed-off-by: Joel Dice <[email protected]> fix clippy warnings and bench/fuzzing errors Signed-off-by: Joel Dice <[email protected]> revert atomic.wit whitespace change Signed-off-by: Joel Dice <[email protected]> fix build when component-model disabled Signed-off-by: Joel Dice <[email protected]> bless component-macro expected output Signed-off-by: Joel Dice <[email protected]> fix no-std build error Signed-off-by: Joel Dice <[email protected]> fix build with --no-default-features --features runtime,component-model Signed-off-by: Joel Dice <[email protected]> partly fix no-std build It's still broken due to the use of `std::collections::HashMap` in crates/wasmtime/src/runtime/vm/component.rs. I'll address that as part of the work to avoid exposing global task/future/stream/error-context handles to guests. Signed-off-by: Joel Dice <[email protected]> maintain per-instance tables for futures, streams, and error-contexts Signed-off-by: Joel Dice <[email protected]> refactor task/stream/future handle lifting/lowering This addresses a couple of issues: - Previously, we were passing task/stream/future/error-context reps directly to instances while keeping track of which instance had access to which rep. That worked fine in that there was no way to forge access to inaccessible reps, but it leaked information about what other instances were doing. Now we maintain per-instance waitable and error-context tables which map the reps to and from the handles which the instance sees. - The `no_std` build was broken due to use of `HashMap` in `runtime::vm::component`, which is now fixed. Note that we use one single table per instance for all tasks, streams, and futures. This is partly necessary because, when async events are delivered to the guest, it wouldn't have enough context to know which stream or future we're talking about if each unique stream and future type had its own table. So at minimum, we need to use the same table for all streams (regardless of payload type), and likewise for futures. Also, per WebAssembly/component-model#395 (comment), the plan is to move towards a shared table for all resource types as well, so this moves us in that direction. Signed-off-by: Joel Dice <[email protected]> fix wave breakage due to new stream/future/error-context types Signed-off-by: Joel Dice <[email protected]> switch wasm-tools to v1.220.0-based branch Signed-off-by: Joel Dice <[email protected]> check `task.return` type at runtime We can't statically verify a given call to `task.return` corresponds to the expected core signature appropriate for the currently running task, so we must do so at runtime. In order to make that check efficient, we intern the types. My initial plan was to use `ModuleInternedTypeIndex` and/or `VMSharedTypeIndex` for interning, but that got hairy with WasmGC considerations, so instead I added new fields to `ComponentTypes` and `ComponentTypesBuilder`. Signed-off-by: Joel Dice <[email protected]> add `TypedFunc::call_concurrent` and refine stream/future APIs This implements what I proposed in https://github.com/dicej/rfcs/blob/component-async/accepted/component-model-async.md#wasmtime. Specifically, it adds: - A new `Promise` type, useful for working with concurrent operations that require access to a `Store` to make progress. - A new `PromisesUnordered` type for `await`ing multiple promises concurrently -`TypedFunc::call_concurrent` (which returns a `Promise`), allowing multiple host->guest calls to run concurrently on the same instance. - Updated `{Stream|Future}{Writer|Reader}` APIs which use `Promise` The upshot is that the embedder can now ergonomically manage arbitrary numbers of concurrent operations. Previously, this was a lot more difficult to do without accidentally starving some of the operations due to another one monopolizing the `Store`. Finally, this includes various refactorings and fixes for bugs exposed by the newer, more versatile APIs. Signed-off-by: Joel Dice <[email protected]> clean up verbosity in component/func.rs Signed-off-by: Joel Dice <[email protected]> snapshot Signed-off-by: Joel Dice <[email protected]> implement stream/future read/write cancellation This required a somewhat viral addition of `Send` and `Sync` bounds for async host function closure types, unfortunately. Signed-off-by: Joel Dice <[email protected]> add `Func::call_concurrent` and `LinkerInstance::func_new_concurrent` Signed-off-by: Joel Dice <[email protected]> dynamic API support for streams/futures/error-contexts Signed-off-by: Joel Dice <[email protected]> support callback-less (AKA stackful) async lifts Signed-off-by: Joel Dice <[email protected]> fix `call_host` regression Signed-off-by: Joel Dice <[email protected]> add component model async end-to-end tests I've ported these over from https://github.com/dicej/component-async-demo Signed-off-by: Joel Dice <[email protected]> fix test regressions and clippy warnings Signed-off-by: Joel Dice <[email protected]> satisfy clippy Signed-off-by: Joel Dice <[email protected]> fix async tests when `component-model-async` enabled Enabling this feature for all tests revealed various missing pieces in the new `concurrent.rs` fiber mechanism, which I've addressed. This adds a bunch of ugly `#[cfg(feature = "component-model-async")]` guards, but those will all go away once I unify the two async fiber implementations. Signed-off-by: Joel Dice <[email protected]> add and modify tests to cover concurrent APIs Primarily, this tests and implements cases where parameters and/or results must be passed via linear memory instead of the stack. Signed-off-by: Joel Dice <[email protected]> `concurrent_{imports|exports}` component macro codegen tests This enables codegen testing of the `concurrent_imports` and `concurrent_exports` options to `wasmtime::component::bindgen` and also fixes code generation for world-level function and resource exports that use the concurrent call style. Signed-off-by: Joel Dice <[email protected]> `concurrent_{imports|exports}` component macro expanded tests This enables testing of the `concurrent_imports` and `concurrent_exports` options in `crates/component-macro/tests/expanded.rs`. Signed-off-by: Joel Dice <[email protected]> add tests/misc_testsuite/component-model-async/*.wast These only test instantiation of components which use various async options and built-ins so far. Next, I'll happy and sad path tests which actually execute code. Signed-off-by: Joel Dice <[email protected]> appease clippy Signed-off-by: Joel Dice <[email protected]> add tests/misc_testsuite/component-model-async/fused.wast Signed-off-by: Joel Dice <[email protected]> add non-panicking bounds checks where appropriate Signed-off-by: Joel Dice <[email protected]> remove post-return bits from async result lift code ...at least until we've determined whether post-return options even make sense for async-lifted exports. Signed-off-by: Joel Dice <[email protected]> fix component-model-async/fused.wast test failure Signed-off-by: Joel Dice <[email protected]> use `enum` types to represent status and event codes Signed-off-by: Joel Dice <[email protected]> fix component-model-async/fused.wast test failure (2nd try) Signed-off-by: Joel Dice <[email protected]> use `gc_types = true` in component-model-async/fused.wast We use `Instruction::RefFunc` when generating adapters for async lifts and/or lowers, which Winch doesn't understand, and apparently `gc_types = true` is what tells the test infra not to use Winch. Signed-off-by: Joel Dice <[email protected]> trap if async function finishes without calling `task.return` Signed-off-by: Joel Dice <[email protected]> update wit-bindgen and fix rebase damage Signed-off-by: Joel Dice <[email protected]> call post-return function if any for async->sync fused calls Signed-off-by: Joel Dice <[email protected]> fix non-component-model-async build; appease clippy Signed-off-by: Joel Dice <[email protected]> bless bindgen output whitespace changes Signed-off-by: Joel Dice <[email protected]> enforce resource borrow requirements for async calls Signed-off-by: Joel Dice <[email protected]> update `wit-bindgen` and simplify `async_borrowing_callee` test Signed-off-by: Joel Dice <[email protected]> call `InstanceFlags::set_may_enter` where appropriate There's still more work to do to fully implement (and test) the reentrance rules for concurrent tasks, but this is a start. Signed-off-by: Joel Dice <[email protected]> finish implementing reentrance checks Signed-off-by: Joel Dice <[email protected]> feat: implement error-context (#1) * feat: initial error-context implementation This commit implements error-context related functions inside the VM, along with tests to ensure that basic error-context.new and error-context.debug-message functionality works. Signed-off-by: Victor Adossi <[email protected]> * wip: add test for error context callee/caller transfer Signed-off-by: Victor Adossi <[email protected]> * wip: test for async context transfer Signed-off-by: Victor Adossi <[email protected]> --------- Signed-off-by: Victor Adossi <[email protected]> run cargo fmt Signed-off-by: Joel Dice <[email protected]> appease clippy Signed-off-by: Joel Dice <[email protected]> pull in Roman's unit stream work; add world-level export test Signed-off-by: Joel Dice <[email protected]>

This adds support for loading, compiling, linking, and running components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) along with the [`stream`, `future`, and `error-context`](WebAssembly/component-model#405) types. It also adds support for generating host bindings such that multiple host functions can be run concurrently with guest tasks -- without monopolizing the `Store`. See the [implementation RFC](bytecodealliance/rfcs#38) for details, as well as [this repo](https://github.com/dicej/component-async-demo) containing end-to-end smoke tests. Signed-off-by: Joel Dice <[email protected]> fix clippy warnings and bench/fuzzing errors Signed-off-by: Joel Dice <[email protected]> revert atomic.wit whitespace change Signed-off-by: Joel Dice <[email protected]> fix build when component-model disabled Signed-off-by: Joel Dice <[email protected]> bless component-macro expected output Signed-off-by: Joel Dice <[email protected]> fix no-std build error Signed-off-by: Joel Dice <[email protected]> fix build with --no-default-features --features runtime,component-model Signed-off-by: Joel Dice <[email protected]> partly fix no-std build It's still broken due to the use of `std::collections::HashMap` in crates/wasmtime/src/runtime/vm/component.rs. I'll address that as part of the work to avoid exposing global task/future/stream/error-context handles to guests. Signed-off-by: Joel Dice <[email protected]> maintain per-instance tables for futures, streams, and error-contexts Signed-off-by: Joel Dice <[email protected]> refactor task/stream/future handle lifting/lowering This addresses a couple of issues: - Previously, we were passing task/stream/future/error-context reps directly to instances while keeping track of which instance had access to which rep. That worked fine in that there was no way to forge access to inaccessible reps, but it leaked information about what other instances were doing. Now we maintain per-instance waitable and error-context tables which map the reps to and from the handles which the instance sees. - The `no_std` build was broken due to use of `HashMap` in `runtime::vm::component`, which is now fixed. Note that we use one single table per instance for all tasks, streams, and futures. This is partly necessary because, when async events are delivered to the guest, it wouldn't have enough context to know which stream or future we're talking about if each unique stream and future type had its own table. So at minimum, we need to use the same table for all streams (regardless of payload type), and likewise for futures. Also, per WebAssembly/component-model#395 (comment), the plan is to move towards a shared table for all resource types as well, so this moves us in that direction. Signed-off-by: Joel Dice <[email protected]> fix wave breakage due to new stream/future/error-context types Signed-off-by: Joel Dice <[email protected]> switch wasm-tools to v1.220.0-based branch Signed-off-by: Joel Dice <[email protected]> check `task.return` type at runtime We can't statically verify a given call to `task.return` corresponds to the expected core signature appropriate for the currently running task, so we must do so at runtime. In order to make that check efficient, we intern the types. My initial plan was to use `ModuleInternedTypeIndex` and/or `VMSharedTypeIndex` for interning, but that got hairy with WasmGC considerations, so instead I added new fields to `ComponentTypes` and `ComponentTypesBuilder`. Signed-off-by: Joel Dice <[email protected]> add `TypedFunc::call_concurrent` and refine stream/future APIs This implements what I proposed in https://github.com/dicej/rfcs/blob/component-async/accepted/component-model-async.md#wasmtime. Specifically, it adds: - A new `Promise` type, useful for working with concurrent operations that require access to a `Store` to make progress. - A new `PromisesUnordered` type for `await`ing multiple promises concurrently -`TypedFunc::call_concurrent` (which returns a `Promise`), allowing multiple host->guest calls to run concurrently on the same instance. - Updated `{Stream|Future}{Writer|Reader}` APIs which use `Promise` The upshot is that the embedder can now ergonomically manage arbitrary numbers of concurrent operations. Previously, this was a lot more difficult to do without accidentally starving some of the operations due to another one monopolizing the `Store`. Finally, this includes various refactorings and fixes for bugs exposed by the newer, more versatile APIs. Signed-off-by: Joel Dice <[email protected]> clean up verbosity in component/func.rs Signed-off-by: Joel Dice <[email protected]> snapshot Signed-off-by: Joel Dice <[email protected]> implement stream/future read/write cancellation This required a somewhat viral addition of `Send` and `Sync` bounds for async host function closure types, unfortunately. Signed-off-by: Joel Dice <[email protected]> add `Func::call_concurrent` and `LinkerInstance::func_new_concurrent` Signed-off-by: Joel Dice <[email protected]> dynamic API support for streams/futures/error-contexts Signed-off-by: Joel Dice <[email protected]> support callback-less (AKA stackful) async lifts Signed-off-by: Joel Dice <[email protected]> fix `call_host` regression Signed-off-by: Joel Dice <[email protected]> add component model async end-to-end tests I've ported these over from https://github.com/dicej/component-async-demo Signed-off-by: Joel Dice <[email protected]> fix test regressions and clippy warnings Signed-off-by: Joel Dice <[email protected]> satisfy clippy Signed-off-by: Joel Dice <[email protected]> fix async tests when `component-model-async` enabled Enabling this feature for all tests revealed various missing pieces in the new `concurrent.rs` fiber mechanism, which I've addressed. This adds a bunch of ugly `#[cfg(feature = "component-model-async")]` guards, but those will all go away once I unify the two async fiber implementations. Signed-off-by: Joel Dice <[email protected]> add and modify tests to cover concurrent APIs Primarily, this tests and implements cases where parameters and/or results must be passed via linear memory instead of the stack. Signed-off-by: Joel Dice <[email protected]> `concurrent_{imports|exports}` component macro codegen tests This enables codegen testing of the `concurrent_imports` and `concurrent_exports` options to `wasmtime::component::bindgen` and also fixes code generation for world-level function and resource exports that use the concurrent call style. Signed-off-by: Joel Dice <[email protected]> `concurrent_{imports|exports}` component macro expanded tests This enables testing of the `concurrent_imports` and `concurrent_exports` options in `crates/component-macro/tests/expanded.rs`. Signed-off-by: Joel Dice <[email protected]> add tests/misc_testsuite/component-model-async/*.wast These only test instantiation of components which use various async options and built-ins so far. Next, I'll happy and sad path tests which actually execute code. Signed-off-by: Joel Dice <[email protected]> appease clippy Signed-off-by: Joel Dice <[email protected]> add tests/misc_testsuite/component-model-async/fused.wast Signed-off-by: Joel Dice <[email protected]> add non-panicking bounds checks where appropriate Signed-off-by: Joel Dice <[email protected]> remove post-return bits from async result lift code ...at least until we've determined whether post-return options even make sense for async-lifted exports. Signed-off-by: Joel Dice <[email protected]> fix component-model-async/fused.wast test failure Signed-off-by: Joel Dice <[email protected]> use `enum` types to represent status and event codes Signed-off-by: Joel Dice <[email protected]> fix component-model-async/fused.wast test failure (2nd try) Signed-off-by: Joel Dice <[email protected]> use `gc_types = true` in component-model-async/fused.wast We use `Instruction::RefFunc` when generating adapters for async lifts and/or lowers, which Winch doesn't understand, and apparently `gc_types = true` is what tells the test infra not to use Winch. Signed-off-by: Joel Dice <[email protected]> trap if async function finishes without calling `task.return` Signed-off-by: Joel Dice <[email protected]> update wit-bindgen and fix rebase damage Signed-off-by: Joel Dice <[email protected]> call post-return function if any for async->sync fused calls Signed-off-by: Joel Dice <[email protected]> fix non-component-model-async build; appease clippy Signed-off-by: Joel Dice <[email protected]> bless bindgen output whitespace changes Signed-off-by: Joel Dice <[email protected]> enforce resource borrow requirements for async calls Signed-off-by: Joel Dice <[email protected]> update `wit-bindgen` and simplify `async_borrowing_callee` test Signed-off-by: Joel Dice <[email protected]> call `InstanceFlags::set_may_enter` where appropriate There's still more work to do to fully implement (and test) the reentrance rules for concurrent tasks, but this is a start. Signed-off-by: Joel Dice <[email protected]> finish implementing reentrance checks Signed-off-by: Joel Dice <[email protected]> feat: implement error-context (#1) * feat: initial error-context implementation This commit implements error-context related functions inside the VM, along with tests to ensure that basic error-context.new and error-context.debug-message functionality works. Signed-off-by: Victor Adossi <[email protected]> * wip: add test for error context callee/caller transfer Signed-off-by: Victor Adossi <[email protected]> * wip: test for async context transfer Signed-off-by: Victor Adossi <[email protected]> --------- Signed-off-by: Victor Adossi <[email protected]> run cargo fmt Signed-off-by: Joel Dice <[email protected]> appease clippy Signed-off-by: Joel Dice <[email protected]> pull in Roman's unit stream work; add world-level export test Signed-off-by: Joel Dice <[email protected]> add unit stream tests Signed-off-by: Joel Dice <[email protected]>

This adds support for loading, compiling, linking, and running components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) along with the [`stream`, `future`, and `error-context`](WebAssembly/component-model#405) types. It also adds support for generating host bindings such that multiple host functions can be run concurrently with guest tasks -- without monopolizing the `Store`. See the [implementation RFC](bytecodealliance/rfcs#38) for details, as well as [this repo](https://github.com/dicej/component-async-demo) containing end-to-end smoke tests. Signed-off-by: Joel Dice <[email protected]> fix clippy warnings and bench/fuzzing errors Signed-off-by: Joel Dice <[email protected]> revert atomic.wit whitespace change Signed-off-by: Joel Dice <[email protected]> fix build when component-model disabled Signed-off-by: Joel Dice <[email protected]> bless component-macro expected output Signed-off-by: Joel Dice <[email protected]> fix no-std build error Signed-off-by: Joel Dice <[email protected]> fix build with --no-default-features --features runtime,component-model Signed-off-by: Joel Dice <[email protected]> partly fix no-std build It's still broken due to the use of `std::collections::HashMap` in crates/wasmtime/src/runtime/vm/component.rs. I'll address that as part of the work to avoid exposing global task/future/stream/error-context handles to guests. Signed-off-by: Joel Dice <[email protected]> maintain per-instance tables for futures, streams, and error-contexts Signed-off-by: Joel Dice <[email protected]> refactor task/stream/future handle lifting/lowering This addresses a couple of issues: - Previously, we were passing task/stream/future/error-context reps directly to instances while keeping track of which instance had access to which rep. That worked fine in that there was no way to forge access to inaccessible reps, but it leaked information about what other instances were doing. Now we maintain per-instance waitable and error-context tables which map the reps to and from the handles which the instance sees. - The `no_std` build was broken due to use of `HashMap` in `runtime::vm::component`, which is now fixed. Note that we use one single table per instance for all tasks, streams, and futures. This is partly necessary because, when async events are delivered to the guest, it wouldn't have enough context to know which stream or future we're talking about if each unique stream and future type had its own table. So at minimum, we need to use the same table for all streams (regardless of payload type), and likewise for futures. Also, per WebAssembly/component-model#395 (comment), the plan is to move towards a shared table for all resource types as well, so this moves us in that direction. Signed-off-by: Joel Dice <[email protected]> fix wave breakage due to new stream/future/error-context types Signed-off-by: Joel Dice <[email protected]> switch wasm-tools to v1.220.0-based branch Signed-off-by: Joel Dice <[email protected]> check `task.return` type at runtime We can't statically verify a given call to `task.return` corresponds to the expected core signature appropriate for the currently running task, so we must do so at runtime. In order to make that check efficient, we intern the types. My initial plan was to use `ModuleInternedTypeIndex` and/or `VMSharedTypeIndex` for interning, but that got hairy with WasmGC considerations, so instead I added new fields to `ComponentTypes` and `ComponentTypesBuilder`. Signed-off-by: Joel Dice <[email protected]> add `TypedFunc::call_concurrent` and refine stream/future APIs This implements what I proposed in https://github.com/dicej/rfcs/blob/component-async/accepted/component-model-async.md#wasmtime. Specifically, it adds: - A new `Promise` type, useful for working with concurrent operations that require access to a `Store` to make progress. - A new `PromisesUnordered` type for `await`ing multiple promises concurrently -`TypedFunc::call_concurrent` (which returns a `Promise`), allowing multiple host->guest calls to run concurrently on the same instance. - Updated `{Stream|Future}{Writer|Reader}` APIs which use `Promise` The upshot is that the embedder can now ergonomically manage arbitrary numbers of concurrent operations. Previously, this was a lot more difficult to do without accidentally starving some of the operations due to another one monopolizing the `Store`. Finally, this includes various refactorings and fixes for bugs exposed by the newer, more versatile APIs. Signed-off-by: Joel Dice <[email protected]> clean up verbosity in component/func.rs Signed-off-by: Joel Dice <[email protected]> snapshot Signed-off-by: Joel Dice <[email protected]> implement stream/future read/write cancellation This required a somewhat viral addition of `Send` and `Sync` bounds for async host function closure types, unfortunately. Signed-off-by: Joel Dice <[email protected]> add `Func::call_concurrent` and `LinkerInstance::func_new_concurrent` Signed-off-by: Joel Dice <[email protected]> dynamic API support for streams/futures/error-contexts Signed-off-by: Joel Dice <[email protected]> support callback-less (AKA stackful) async lifts Signed-off-by: Joel Dice <[email protected]> fix `call_host` regression Signed-off-by: Joel Dice <[email protected]> add component model async end-to-end tests I've ported these over from https://github.com/dicej/component-async-demo Signed-off-by: Joel Dice <[email protected]> fix test regressions and clippy warnings Signed-off-by: Joel Dice <[email protected]> satisfy clippy Signed-off-by: Joel Dice <[email protected]> fix async tests when `component-model-async` enabled Enabling this feature for all tests revealed various missing pieces in the new `concurrent.rs` fiber mechanism, which I've addressed. This adds a bunch of ugly `#[cfg(feature = "component-model-async")]` guards, but those will all go away once I unify the two async fiber implementations. Signed-off-by: Joel Dice <[email protected]> add and modify tests to cover concurrent APIs Primarily, this tests and implements cases where parameters and/or results must be passed via linear memory instead of the stack. Signed-off-by: Joel Dice <[email protected]> `concurrent_{imports|exports}` component macro codegen tests This enables codegen testing of the `concurrent_imports` and `concurrent_exports` options to `wasmtime::component::bindgen` and also fixes code generation for world-level function and resource exports that use the concurrent call style. Signed-off-by: Joel Dice <[email protected]> `concurrent_{imports|exports}` component macro expanded tests This enables testing of the `concurrent_imports` and `concurrent_exports` options in `crates/component-macro/tests/expanded.rs`. Signed-off-by: Joel Dice <[email protected]> add tests/misc_testsuite/component-model-async/*.wast These only test instantiation of components which use various async options and built-ins so far. Next, I'll happy and sad path tests which actually execute code. Signed-off-by: Joel Dice <[email protected]> appease clippy Signed-off-by: Joel Dice <[email protected]> add tests/misc_testsuite/component-model-async/fused.wast Signed-off-by: Joel Dice <[email protected]> add non-panicking bounds checks where appropriate Signed-off-by: Joel Dice <[email protected]> remove post-return bits from async result lift code ...at least until we've determined whether post-return options even make sense for async-lifted exports. Signed-off-by: Joel Dice <[email protected]> fix component-model-async/fused.wast test failure Signed-off-by: Joel Dice <[email protected]> use `enum` types to represent status and event codes Signed-off-by: Joel Dice <[email protected]> fix component-model-async/fused.wast test failure (2nd try) Signed-off-by: Joel Dice <[email protected]> use `gc_types = true` in component-model-async/fused.wast We use `Instruction::RefFunc` when generating adapters for async lifts and/or lowers, which Winch doesn't understand, and apparently `gc_types = true` is what tells the test infra not to use Winch. Signed-off-by: Joel Dice <[email protected]> trap if async function finishes without calling `task.return` Signed-off-by: Joel Dice <[email protected]> update wit-bindgen and fix rebase damage Signed-off-by: Joel Dice <[email protected]> call post-return function if any for async->sync fused calls Signed-off-by: Joel Dice <[email protected]> fix non-component-model-async build; appease clippy Signed-off-by: Joel Dice <[email protected]> bless bindgen output whitespace changes Signed-off-by: Joel Dice <[email protected]> enforce resource borrow requirements for async calls Signed-off-by: Joel Dice <[email protected]> update `wit-bindgen` and simplify `async_borrowing_callee` test Signed-off-by: Joel Dice <[email protected]> call `InstanceFlags::set_may_enter` where appropriate There's still more work to do to fully implement (and test) the reentrance rules for concurrent tasks, but this is a start. Signed-off-by: Joel Dice <[email protected]> finish implementing reentrance checks Signed-off-by: Joel Dice <[email protected]> feat: implement error-context (bytecodealliance#1) * feat: initial error-context implementation This commit implements error-context related functions inside the VM, along with tests to ensure that basic error-context.new and error-context.debug-message functionality works. Signed-off-by: Victor Adossi <[email protected]> * wip: add test for error context callee/caller transfer Signed-off-by: Victor Adossi <[email protected]> * wip: test for async context transfer Signed-off-by: Victor Adossi <[email protected]> --------- Signed-off-by: Victor Adossi <[email protected]> run cargo fmt Signed-off-by: Joel Dice <[email protected]> appease clippy Signed-off-by: Joel Dice <[email protected]> pull in Roman's unit stream work; add world-level export test Signed-off-by: Joel Dice <[email protected]> add unit stream tests Signed-off-by: Joel Dice <[email protected]>

This adds support for loading, compiling, linking, and running components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) along with the [`stream`, `future`, and `error-context`](WebAssembly/component-model#405) types. It also adds support for generating host bindings such that multiple host functions can be run concurrently with guest tasks -- without monopolizing the `Store`. See the [implementation RFC](bytecodealliance/rfcs#38) for details, as well as [this repo](https://github.com/dicej/component-async-demo) containing end-to-end smoke tests. Signed-off-by: Joel Dice <[email protected]> fix clippy warnings and bench/fuzzing errors Signed-off-by: Joel Dice <[email protected]> revert atomic.wit whitespace change Signed-off-by: Joel Dice <[email protected]> fix build when component-model disabled Signed-off-by: Joel Dice <[email protected]> bless component-macro expected output Signed-off-by: Joel Dice <[email protected]> fix no-std build error Signed-off-by: Joel Dice <[email protected]> fix build with --no-default-features --features runtime,component-model Signed-off-by: Joel Dice <[email protected]> partly fix no-std build It's still broken due to the use of `std::collections::HashMap` in crates/wasmtime/src/runtime/vm/component.rs. I'll address that as part of the work to avoid exposing global task/future/stream/error-context handles to guests. Signed-off-by: Joel Dice <[email protected]> maintain per-instance tables for futures, streams, and error-contexts Signed-off-by: Joel Dice <[email protected]> refactor task/stream/future handle lifting/lowering This addresses a couple of issues: - Previously, we were passing task/stream/future/error-context reps directly to instances while keeping track of which instance had access to which rep. That worked fine in that there was no way to forge access to inaccessible reps, but it leaked information about what other instances were doing. Now we maintain per-instance waitable and error-context tables which map the reps to and from the handles which the instance sees. - The `no_std` build was broken due to use of `HashMap` in `runtime::vm::component`, which is now fixed. Note that we use one single table per instance for all tasks, streams, and futures. This is partly necessary because, when async events are delivered to the guest, it wouldn't have enough context to know which stream or future we're talking about if each unique stream and future type had its own table. So at minimum, we need to use the same table for all streams (regardless of payload type), and likewise for futures. Also, per WebAssembly/component-model#395 (comment), the plan is to move towards a shared table for all resource types as well, so this moves us in that direction. Signed-off-by: Joel Dice <[email protected]> fix wave breakage due to new stream/future/error-context types Signed-off-by: Joel Dice <[email protected]> switch wasm-tools to v1.220.0-based branch Signed-off-by: Joel Dice <[email protected]> check `task.return` type at runtime We can't statically verify a given call to `task.return` corresponds to the expected core signature appropriate for the currently running task, so we must do so at runtime. In order to make that check efficient, we intern the types. My initial plan was to use `ModuleInternedTypeIndex` and/or `VMSharedTypeIndex` for interning, but that got hairy with WasmGC considerations, so instead I added new fields to `ComponentTypes` and `ComponentTypesBuilder`. Signed-off-by: Joel Dice <[email protected]> add `TypedFunc::call_concurrent` and refine stream/future APIs This implements what I proposed in https://github.com/dicej/rfcs/blob/component-async/accepted/component-model-async.md#wasmtime. Specifically, it adds: - A new `Promise` type, useful for working with concurrent operations that require access to a `Store` to make progress. - A new `PromisesUnordered` type for `await`ing multiple promises concurrently -`TypedFunc::call_concurrent` (which returns a `Promise`), allowing multiple host->guest calls to run concurrently on the same instance. - Updated `{Stream|Future}{Writer|Reader}` APIs which use `Promise` The upshot is that the embedder can now ergonomically manage arbitrary numbers of concurrent operations. Previously, this was a lot more difficult to do without accidentally starving some of the operations due to another one monopolizing the `Store`. Finally, this includes various refactorings and fixes for bugs exposed by the newer, more versatile APIs. Signed-off-by: Joel Dice <[email protected]> clean up verbosity in component/func.rs Signed-off-by: Joel Dice <[email protected]> snapshot Signed-off-by: Joel Dice <[email protected]> implement stream/future read/write cancellation This required a somewhat viral addition of `Send` and `Sync` bounds for async host function closure types, unfortunately. Signed-off-by: Joel Dice <[email protected]> add `Func::call_concurrent` and `LinkerInstance::func_new_concurrent` Signed-off-by: Joel Dice <[email protected]> dynamic API support for streams/futures/error-contexts Signed-off-by: Joel Dice <[email protected]> support callback-less (AKA stackful) async lifts Signed-off-by: Joel Dice <[email protected]> fix `call_host` regression Signed-off-by: Joel Dice <[email protected]> add component model async end-to-end tests I've ported these over from https://github.com/dicej/component-async-demo Signed-off-by: Joel Dice <[email protected]> fix test regressions and clippy warnings Signed-off-by: Joel Dice <[email protected]> satisfy clippy Signed-off-by: Joel Dice <[email protected]> fix async tests when `component-model-async` enabled Enabling this feature for all tests revealed various missing pieces in the new `concurrent.rs` fiber mechanism, which I've addressed. This adds a bunch of ugly `#[cfg(feature = "component-model-async")]` guards, but those will all go away once I unify the two async fiber implementations. Signed-off-by: Joel Dice <[email protected]> add and modify tests to cover concurrent APIs Primarily, this tests and implements cases where parameters and/or results must be passed via linear memory instead of the stack. Signed-off-by: Joel Dice <[email protected]> `concurrent_{imports|exports}` component macro codegen tests This enables codegen testing of the `concurrent_imports` and `concurrent_exports` options to `wasmtime::component::bindgen` and also fixes code generation for world-level function and resource exports that use the concurrent call style. Signed-off-by: Joel Dice <[email protected]> `concurrent_{imports|exports}` component macro expanded tests This enables testing of the `concurrent_imports` and `concurrent_exports` options in `crates/component-macro/tests/expanded.rs`. Signed-off-by: Joel Dice <[email protected]> add tests/misc_testsuite/component-model-async/*.wast These only test instantiation of components which use various async options and built-ins so far. Next, I'll happy and sad path tests which actually execute code. Signed-off-by: Joel Dice <[email protected]> appease clippy Signed-off-by: Joel Dice <[email protected]> add tests/misc_testsuite/component-model-async/fused.wast Signed-off-by: Joel Dice <[email protected]> add non-panicking bounds checks where appropriate Signed-off-by: Joel Dice <[email protected]> remove post-return bits from async result lift code ...at least until we've determined whether post-return options even make sense for async-lifted exports. Signed-off-by: Joel Dice <[email protected]> fix component-model-async/fused.wast test failure Signed-off-by: Joel Dice <[email protected]> use `enum` types to represent status and event codes Signed-off-by: Joel Dice <[email protected]> fix component-model-async/fused.wast test failure (2nd try) Signed-off-by: Joel Dice <[email protected]> use `gc_types = true` in component-model-async/fused.wast We use `Instruction::RefFunc` when generating adapters for async lifts and/or lowers, which Winch doesn't understand, and apparently `gc_types = true` is what tells the test infra not to use Winch. Signed-off-by: Joel Dice <[email protected]> trap if async function finishes without calling `task.return` Signed-off-by: Joel Dice <[email protected]> update wit-bindgen and fix rebase damage Signed-off-by: Joel Dice <[email protected]> call post-return function if any for async->sync fused calls Signed-off-by: Joel Dice <[email protected]> fix non-component-model-async build; appease clippy Signed-off-by: Joel Dice <[email protected]> bless bindgen output whitespace changes Signed-off-by: Joel Dice <[email protected]> enforce resource borrow requirements for async calls Signed-off-by: Joel Dice <[email protected]> update `wit-bindgen` and simplify `async_borrowing_callee` test Signed-off-by: Joel Dice <[email protected]> call `InstanceFlags::set_may_enter` where appropriate There's still more work to do to fully implement (and test) the reentrance rules for concurrent tasks, but this is a start. Signed-off-by: Joel Dice <[email protected]> finish implementing reentrance checks Signed-off-by: Joel Dice <[email protected]> feat: implement error-context (#1) * feat: initial error-context implementation This commit implements error-context related functions inside the VM, along with tests to ensure that basic error-context.new and error-context.debug-message functionality works. Signed-off-by: Victor Adossi <[email protected]> * wip: add test for error context callee/caller transfer Signed-off-by: Victor Adossi <[email protected]> * wip: test for async context transfer Signed-off-by: Victor Adossi <[email protected]> --------- Signed-off-by: Victor Adossi <[email protected]> run cargo fmt Signed-off-by: Joel Dice <[email protected]> appease clippy Signed-off-by: Joel Dice <[email protected]> pull in Roman's unit stream work; add world-level export test Signed-off-by: Joel Dice <[email protected]> add unit stream tests Signed-off-by: Joel Dice <[email protected]>

This adds support for loading, compiling, linking, and running components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) along with the [`stream`, `future`, and `error-context`](WebAssembly/component-model#405) types. It also adds support for generating host bindings such that multiple host functions can be run concurrently with guest tasks -- without monopolizing the `Store`. See the [implementation RFC](bytecodealliance/rfcs#38) for details, as well as [this repo](https://github.com/dicej/component-async-demo) containing end-to-end smoke tests. Signed-off-by: Joel Dice <[email protected]> fix clippy warnings and bench/fuzzing errors Signed-off-by: Joel Dice <[email protected]> revert atomic.wit whitespace change Signed-off-by: Joel Dice <[email protected]> fix build when component-model disabled Signed-off-by: Joel Dice <[email protected]> bless component-macro expected output Signed-off-by: Joel Dice <[email protected]> fix no-std build error Signed-off-by: Joel Dice <[email protected]> fix build with --no-default-features --features runtime,component-model Signed-off-by: Joel Dice <[email protected]> partly fix no-std build It's still broken due to the use of `std::collections::HashMap` in crates/wasmtime/src/runtime/vm/component.rs. I'll address that as part of the work to avoid exposing global task/future/stream/error-context handles to guests. Signed-off-by: Joel Dice <[email protected]> maintain per-instance tables for futures, streams, and error-contexts Signed-off-by: Joel Dice <[email protected]> refactor task/stream/future handle lifting/lowering This addresses a couple of issues: - Previously, we were passing task/stream/future/error-context reps directly to instances while keeping track of which instance had access to which rep. That worked fine in that there was no way to forge access to inaccessible reps, but it leaked information about what other instances were doing. Now we maintain per-instance waitable and error-context tables which map the reps to and from the handles which the instance sees. - The `no_std` build was broken due to use of `HashMap` in `runtime::vm::component`, which is now fixed. Note that we use one single table per instance for all tasks, streams, and futures. This is partly necessary because, when async events are delivered to the guest, it wouldn't have enough context to know which stream or future we're talking about if each unique stream and future type had its own table. So at minimum, we need to use the same table for all streams (regardless of payload type), and likewise for futures. Also, per WebAssembly/component-model#395 (comment), the plan is to move towards a shared table for all resource types as well, so this moves us in that direction. Signed-off-by: Joel Dice <[email protected]> fix wave breakage due to new stream/future/error-context types Signed-off-by: Joel Dice <[email protected]> switch wasm-tools to v1.220.0-based branch Signed-off-by: Joel Dice <[email protected]> check `task.return` type at runtime We can't statically verify a given call to `task.return` corresponds to the expected core signature appropriate for the currently running task, so we must do so at runtime. In order to make that check efficient, we intern the types. My initial plan was to use `ModuleInternedTypeIndex` and/or `VMSharedTypeIndex` for interning, but that got hairy with WasmGC considerations, so instead I added new fields to `ComponentTypes` and `ComponentTypesBuilder`. Signed-off-by: Joel Dice <[email protected]> add `TypedFunc::call_concurrent` and refine stream/future APIs This implements what I proposed in https://github.com/dicej/rfcs/blob/component-async/accepted/component-model-async.md#wasmtime. Specifically, it adds: - A new `Promise` type, useful for working with concurrent operations that require access to a `Store` to make progress. - A new `PromisesUnordered` type for `await`ing multiple promises concurrently -`TypedFunc::call_concurrent` (which returns a `Promise`), allowing multiple host->guest calls to run concurrently on the same instance. - Updated `{Stream|Future}{Writer|Reader}` APIs which use `Promise` The upshot is that the embedder can now ergonomically manage arbitrary numbers of concurrent operations. Previously, this was a lot more difficult to do without accidentally starving some of the operations due to another one monopolizing the `Store`. Finally, this includes various refactorings and fixes for bugs exposed by the newer, more versatile APIs. Signed-off-by: Joel Dice <[email protected]> clean up verbosity in component/func.rs Signed-off-by: Joel Dice <[email protected]> snapshot Signed-off-by: Joel Dice <[email protected]> implement stream/future read/write cancellation This required a somewhat viral addition of `Send` and `Sync` bounds for async host function closure types, unfortunately. Signed-off-by: Joel Dice <[email protected]> add `Func::call_concurrent` and `LinkerInstance::func_new_concurrent` Signed-off-by: Joel Dice <[email protected]> dynamic API support for streams/futures/error-contexts Signed-off-by: Joel Dice <[email protected]> support callback-less (AKA stackful) async lifts Signed-off-by: Joel Dice <[email protected]> fix `call_host` regression Signed-off-by: Joel Dice <[email protected]> add component model async end-to-end tests I've ported these over from https://github.com/dicej/component-async-demo Signed-off-by: Joel Dice <[email protected]> fix test regressions and clippy warnings Signed-off-by: Joel Dice <[email protected]> satisfy clippy Signed-off-by: Joel Dice <[email protected]> fix async tests when `component-model-async` enabled Enabling this feature for all tests revealed various missing pieces in the new `concurrent.rs` fiber mechanism, which I've addressed. This adds a bunch of ugly `#[cfg(feature = "component-model-async")]` guards, but those will all go away once I unify the two async fiber implementations. Signed-off-by: Joel Dice <[email protected]> add and modify tests to cover concurrent APIs Primarily, this tests and implements cases where parameters and/or results must be passed via linear memory instead of the stack. Signed-off-by: Joel Dice <[email protected]> `concurrent_{imports|exports}` component macro codegen tests This enables codegen testing of the `concurrent_imports` and `concurrent_exports` options to `wasmtime::component::bindgen` and also fixes code generation for world-level function and resource exports that use the concurrent call style. Signed-off-by: Joel Dice <[email protected]> `concurrent_{imports|exports}` component macro expanded tests This enables testing of the `concurrent_imports` and `concurrent_exports` options in `crates/component-macro/tests/expanded.rs`. Signed-off-by: Joel Dice <[email protected]> add tests/misc_testsuite/component-model-async/*.wast These only test instantiation of components which use various async options and built-ins so far. Next, I'll happy and sad path tests which actually execute code. Signed-off-by: Joel Dice <[email protected]> appease clippy Signed-off-by: Joel Dice <[email protected]> add tests/misc_testsuite/component-model-async/fused.wast Signed-off-by: Joel Dice <[email protected]> add non-panicking bounds checks where appropriate Signed-off-by: Joel Dice <[email protected]> remove post-return bits from async result lift code ...at least until we've determined whether post-return options even make sense for async-lifted exports. Signed-off-by: Joel Dice <[email protected]> fix component-model-async/fused.wast test failure Signed-off-by: Joel Dice <[email protected]> use `enum` types to represent status and event codes Signed-off-by: Joel Dice <[email protected]> fix component-model-async/fused.wast test failure (2nd try) Signed-off-by: Joel Dice <[email protected]> use `gc_types = true` in component-model-async/fused.wast We use `Instruction::RefFunc` when generating adapters for async lifts and/or lowers, which Winch doesn't understand, and apparently `gc_types = true` is what tells the test infra not to use Winch. Signed-off-by: Joel Dice <[email protected]> trap if async function finishes without calling `task.return` Signed-off-by: Joel Dice <[email protected]> update wit-bindgen and fix rebase damage Signed-off-by: Joel Dice <[email protected]> call post-return function if any for async->sync fused calls Signed-off-by: Joel Dice <[email protected]> fix non-component-model-async build; appease clippy Signed-off-by: Joel Dice <[email protected]> bless bindgen output whitespace changes Signed-off-by: Joel Dice <[email protected]> enforce resource borrow requirements for async calls Signed-off-by: Joel Dice <[email protected]> update `wit-bindgen` and simplify `async_borrowing_callee` test Signed-off-by: Joel Dice <[email protected]> call `InstanceFlags::set_may_enter` where appropriate There's still more work to do to fully implement (and test) the reentrance rules for concurrent tasks, but this is a start. Signed-off-by: Joel Dice <[email protected]> finish implementing reentrance checks Signed-off-by: Joel Dice <[email protected]> feat: implement error-context (#1) * feat: initial error-context implementation This commit implements error-context related functions inside the VM, along with tests to ensure that basic error-context.new and error-context.debug-message functionality works. Signed-off-by: Victor Adossi <[email protected]> * wip: add test for error context callee/caller transfer Signed-off-by: Victor Adossi <[email protected]> * wip: test for async context transfer Signed-off-by: Victor Adossi <[email protected]> --------- Signed-off-by: Victor Adossi <[email protected]> run cargo fmt Signed-off-by: Joel Dice <[email protected]> appease clippy Signed-off-by: Joel Dice <[email protected]> pull in Roman's unit stream work; add world-level export test Signed-off-by: Joel Dice <[email protected]> add unit stream tests Signed-off-by: Joel Dice <[email protected]> CI fixes Signed-off-by: Joel Dice <[email protected]>

This is another piece of bytecodealliance#9582 which I'm splitting out to make review easier. This test exercises fused adapter generation for various flavors of intercomponent async->async, async->sync, and sync->async calls. The remaining changes fill in some TODOs to make the test pass. Signed-off-by: Joel Dice <[email protected]> address review feedback Signed-off-by: Joel Dice <[email protected]> update `task.return` to match spec This temporarily switches to my fork of `wasm-tools` until bytecodealliance/wasm-tools#1989 is merged. Signed-off-by: Joel Dice <[email protected]> refactor `component-model-async` cfg attrs per review feedback Also, switch to upstream `wasm-tools` main branch. Signed-off-by: Joel Dice <[email protected]> add `Compiler::new` constructor Signed-off-by: Joel Dice <[email protected]> add comments covering various lower/lift combos in FACT generator Signed-off-by: Joel Dice <[email protected]> add `Compiler::global_set` helper function Signed-off-by: Joel Dice <[email protected]> add component-model-async/task-builtins.wast test This is another piece of bytecodealliance#9582 which I'm splitting out to make review easier. This test includes components which use the `task.backpressure`, `task.return`, `task.wait`, `task.poll`, `task.yield`, and `subtask.drop` builtins. The rest of the changes fill in some TODOs to make the test pass. Signed-off-by: Joel Dice <[email protected]> Add support for async/streams/futures This adds support for loading, compiling, linking, and running components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) along with the [`stream`, `future`, and `error-context`](WebAssembly/component-model#405) types. It also adds support for generating host bindings such that multiple host functions can be run concurrently with guest tasks -- without monopolizing the `Store`. See the [implementation RFC](bytecodealliance/rfcs#38) for details, as well as [this repo](https://github.com/dicej/component-async-demo) containing end-to-end smoke tests. Signed-off-by: Joel Dice <[email protected]> fix clippy warnings and bench/fuzzing errors Signed-off-by: Joel Dice <[email protected]> revert atomic.wit whitespace change Signed-off-by: Joel Dice <[email protected]> fix build when component-model disabled Signed-off-by: Joel Dice <[email protected]> bless component-macro expected output Signed-off-by: Joel Dice <[email protected]> fix no-std build error Signed-off-by: Joel Dice <[email protected]> fix build with --no-default-features --features runtime,component-model Signed-off-by: Joel Dice <[email protected]> partly fix no-std build It's still broken due to the use of `std::collections::HashMap` in crates/wasmtime/src/runtime/vm/component.rs. I'll address that as part of the work to avoid exposing global task/future/stream/error-context handles to guests. Signed-off-by: Joel Dice <[email protected]> maintain per-instance tables for futures, streams, and error-contexts Signed-off-by: Joel Dice <[email protected]> refactor task/stream/future handle lifting/lowering This addresses a couple of issues: - Previously, we were passing task/stream/future/error-context reps directly to instances while keeping track of which instance had access to which rep. That worked fine in that there was no way to forge access to inaccessible reps, but it leaked information about what other instances were doing. Now we maintain per-instance waitable and error-context tables which map the reps to and from the handles which the instance sees. - The `no_std` build was broken due to use of `HashMap` in `runtime::vm::component`, which is now fixed. Note that we use one single table per instance for all tasks, streams, and futures. This is partly necessary because, when async events are delivered to the guest, it wouldn't have enough context to know which stream or future we're talking about if each unique stream and future type had its own table. So at minimum, we need to use the same table for all streams (regardless of payload type), and likewise for futures. Also, per WebAssembly/component-model#395 (comment), the plan is to move towards a shared table for all resource types as well, so this moves us in that direction. Signed-off-by: Joel Dice <[email protected]> fix wave breakage due to new stream/future/error-context types Signed-off-by: Joel Dice <[email protected]> switch wasm-tools to v1.220.0-based branch Signed-off-by: Joel Dice <[email protected]> check `task.return` type at runtime We can't statically verify a given call to `task.return` corresponds to the expected core signature appropriate for the currently running task, so we must do so at runtime. In order to make that check efficient, we intern the types. My initial plan was to use `ModuleInternedTypeIndex` and/or `VMSharedTypeIndex` for interning, but that got hairy with WasmGC considerations, so instead I added new fields to `ComponentTypes` and `ComponentTypesBuilder`. Signed-off-by: Joel Dice <[email protected]> add `TypedFunc::call_concurrent` and refine stream/future APIs This implements what I proposed in https://github.com/dicej/rfcs/blob/component-async/accepted/component-model-async.md#wasmtime. Specifically, it adds: - A new `Promise` type, useful for working with concurrent operations that require access to a `Store` to make progress. - A new `PromisesUnordered` type for `await`ing multiple promises concurrently -`TypedFunc::call_concurrent` (which returns a `Promise`), allowing multiple host->guest calls to run concurrently on the same instance. - Updated `{Stream|Future}{Writer|Reader}` APIs which use `Promise` The upshot is that the embedder can now ergonomically manage arbitrary numbers of concurrent operations. Previously, this was a lot more difficult to do without accidentally starving some of the operations due to another one monopolizing the `Store`. Finally, this includes various refactorings and fixes for bugs exposed by the newer, more versatile APIs. Signed-off-by: Joel Dice <[email protected]> clean up verbosity in component/func.rs Signed-off-by: Joel Dice <[email protected]> snapshot Signed-off-by: Joel Dice <[email protected]> implement stream/future read/write cancellation This required a somewhat viral addition of `Send` and `Sync` bounds for async host function closure types, unfortunately. Signed-off-by: Joel Dice <[email protected]> add `Func::call_concurrent` and `LinkerInstance::func_new_concurrent` Signed-off-by: Joel Dice <[email protected]> dynamic API support for streams/futures/error-contexts Signed-off-by: Joel Dice <[email protected]> support callback-less (AKA stackful) async lifts Signed-off-by: Joel Dice <[email protected]> fix `call_host` regression Signed-off-by: Joel Dice <[email protected]> add component model async end-to-end tests I've ported these over from https://github.com/dicej/component-async-demo Signed-off-by: Joel Dice <[email protected]> fix test regressions and clippy warnings Signed-off-by: Joel Dice <[email protected]> satisfy clippy Signed-off-by: Joel Dice <[email protected]> fix async tests when `component-model-async` enabled Enabling this feature for all tests revealed various missing pieces in the new `concurrent.rs` fiber mechanism, which I've addressed. This adds a bunch of ugly `#[cfg(feature = "component-model-async")]` guards, but those will all go away once I unify the two async fiber implementations. Signed-off-by: Joel Dice <[email protected]> add and modify tests to cover concurrent APIs Primarily, this tests and implements cases where parameters and/or results must be passed via linear memory instead of the stack. Signed-off-by: Joel Dice <[email protected]> `concurrent_{imports|exports}` component macro codegen tests This enables codegen testing of the `concurrent_imports` and `concurrent_exports` options to `wasmtime::component::bindgen` and also fixes code generation for world-level function and resource exports that use the concurrent call style. Signed-off-by: Joel Dice <[email protected]> `concurrent_{imports|exports}` component macro expanded tests This enables testing of the `concurrent_imports` and `concurrent_exports` options in `crates/component-macro/tests/expanded.rs`. Signed-off-by: Joel Dice <[email protected]> add tests/misc_testsuite/component-model-async/*.wast These only test instantiation of components which use various async options and built-ins so far. Next, I'll happy and sad path tests which actually execute code. Signed-off-by: Joel Dice <[email protected]> appease clippy Signed-off-by: Joel Dice <[email protected]> add tests/misc_testsuite/component-model-async/fused.wast Signed-off-by: Joel Dice <[email protected]> add non-panicking bounds checks where appropriate Signed-off-by: Joel Dice <[email protected]> remove post-return bits from async result lift code ...at least until we've determined whether post-return options even make sense for async-lifted exports. Signed-off-by: Joel Dice <[email protected]> fix component-model-async/fused.wast test failure Signed-off-by: Joel Dice <[email protected]> use `enum` types to represent status and event codes Signed-off-by: Joel Dice <[email protected]> fix component-model-async/fused.wast test failure (2nd try) Signed-off-by: Joel Dice <[email protected]> use `gc_types = true` in component-model-async/fused.wast We use `Instruction::RefFunc` when generating adapters for async lifts and/or lowers, which Winch doesn't understand, and apparently `gc_types = true` is what tells the test infra not to use Winch. Signed-off-by: Joel Dice <[email protected]> trap if async function finishes without calling `task.return` Signed-off-by: Joel Dice <[email protected]> update wit-bindgen and fix rebase damage Signed-off-by: Joel Dice <[email protected]> call post-return function if any for async->sync fused calls Signed-off-by: Joel Dice <[email protected]> fix non-component-model-async build; appease clippy Signed-off-by: Joel Dice <[email protected]> bless bindgen output whitespace changes Signed-off-by: Joel Dice <[email protected]> enforce resource borrow requirements for async calls Signed-off-by: Joel Dice <[email protected]> update `wit-bindgen` and simplify `async_borrowing_callee` test Signed-off-by: Joel Dice <[email protected]> call `InstanceFlags::set_may_enter` where appropriate There's still more work to do to fully implement (and test) the reentrance rules for concurrent tasks, but this is a start. Signed-off-by: Joel Dice <[email protected]> finish implementing reentrance checks Signed-off-by: Joel Dice <[email protected]> feat: implement error-context (#1) * feat: initial error-context implementation This commit implements error-context related functions inside the VM, along with tests to ensure that basic error-context.new and error-context.debug-message functionality works. Signed-off-by: Victor Adossi <[email protected]> * wip: add test for error context callee/caller transfer Signed-off-by: Victor Adossi <[email protected]> * wip: test for async context transfer Signed-off-by: Victor Adossi <[email protected]> --------- Signed-off-by: Victor Adossi <[email protected]> run cargo fmt Signed-off-by: Joel Dice <[email protected]> appease clippy Signed-off-by: Joel Dice <[email protected]> pull in Roman's unit stream work; add world-level export test Signed-off-by: Joel Dice <[email protected]> add unit stream tests Signed-off-by: Joel Dice <[email protected]> CI fixes Signed-off-by: Joel Dice <[email protected]>

This adds support for loading, compiling, linking, and running components which use the [Async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) along with the [`stream`, `future`, and `error-context`](WebAssembly/component-model#405) types. It also adds support for generating host bindings such that multiple host functions can be run concurrently with guest tasks -- without monopolizing the `Store`. See the [implementation RFC](bytecodealliance/rfcs#38) for details, as well as [this repo](https://github.com/dicej/component-async-demo) containing end-to-end smoke tests. Signed-off-by: Joel Dice <[email protected]> fix clippy warnings and bench/fuzzing errors Signed-off-by: Joel Dice <[email protected]> revert atomic.wit whitespace change Signed-off-by: Joel Dice <[email protected]> fix build when component-model disabled Signed-off-by: Joel Dice <[email protected]> bless component-macro expected output Signed-off-by: Joel Dice <[email protected]> fix no-std build error Signed-off-by: Joel Dice <[email protected]> fix build with --no-default-features --features runtime,component-model Signed-off-by: Joel Dice <[email protected]> partly fix no-std build It's still broken due to the use of `std::collections::HashMap` in crates/wasmtime/src/runtime/vm/component.rs. I'll address that as part of the work to avoid exposing global task/future/stream/error-context handles to guests. Signed-off-by: Joel Dice <[email protected]> maintain per-instance tables for futures, streams, and error-contexts Signed-off-by: Joel Dice <[email protected]> refactor task/stream/future handle lifting/lowering This addresses a couple of issues: - Previously, we were passing task/stream/future/error-context reps directly to instances while keeping track of which instance had access to which rep. That worked fine in that there was no way to forge access to inaccessible reps, but it leaked information about what other instances were doing. Now we maintain per-instance waitable and error-context tables which map the reps to and from the handles which the instance sees. - The `no_std` build was broken due to use of `HashMap` in `runtime::vm::component`, which is now fixed. Note that we use one single table per instance for all tasks, streams, and futures. This is partly necessary because, when async events are delivered to the guest, it wouldn't have enough context to know which stream or future we're talking about if each unique stream and future type had its own table. So at minimum, we need to use the same table for all streams (regardless of payload type), and likewise for futures. Also, per WebAssembly/component-model#395 (comment), the plan is to move towards a shared table for all resource types as well, so this moves us in that direction. Signed-off-by: Joel Dice <[email protected]> fix wave breakage due to new stream/future/error-context types Signed-off-by: Joel Dice <[email protected]> switch wasm-tools to v1.220.0-based branch Signed-off-by: Joel Dice <[email protected]> check `task.return` type at runtime We can't statically verify a given call to `task.return` corresponds to the expected core signature appropriate for the currently running task, so we must do so at runtime. In order to make that check efficient, we intern the types. My initial plan was to use `ModuleInternedTypeIndex` and/or `VMSharedTypeIndex` for interning, but that got hairy with WasmGC considerations, so instead I added new fields to `ComponentTypes` and `ComponentTypesBuilder`. Signed-off-by: Joel Dice <[email protected]> add `TypedFunc::call_concurrent` and refine stream/future APIs This implements what I proposed in https://github.com/dicej/rfcs/blob/component-async/accepted/component-model-async.md#wasmtime. Specifically, it adds: - A new `Promise` type, useful for working with concurrent operations that require access to a `Store` to make progress. - A new `PromisesUnordered` type for `await`ing multiple promises concurrently -`TypedFunc::call_concurrent` (which returns a `Promise`), allowing multiple host->guest calls to run concurrently on the same instance. - Updated `{Stream|Future}{Writer|Reader}` APIs which use `Promise` The upshot is that the embedder can now ergonomically manage arbitrary numbers of concurrent operations. Previously, this was a lot more difficult to do without accidentally starving some of the operations due to another one monopolizing the `Store`. Finally, this includes various refactorings and fixes for bugs exposed by the newer, more versatile APIs. Signed-off-by: Joel Dice <[email protected]> clean up verbosity in component/func.rs Signed-off-by: Joel Dice <[email protected]> snapshot Signed-off-by: Joel Dice <[email protected]> implement stream/future read/write cancellation This required a somewhat viral addition of `Send` and `Sync` bounds for async host function closure types, unfortunately. Signed-off-by: Joel Dice <[email protected]> add `Func::call_concurrent` and `LinkerInstance::func_new_concurrent` Signed-off-by: Joel Dice <[email protected]> dynamic API support for streams/futures/error-contexts Signed-off-by: Joel Dice <[email protected]> support callback-less (AKA stackful) async lifts Signed-off-by: Joel Dice <[email protected]> fix `call_host` regression Signed-off-by: Joel Dice <[email protected]> add component model async end-to-end tests I've ported these over from https://github.com/dicej/component-async-demo Signed-off-by: Joel Dice <[email protected]> fix test regressions and clippy warnings Signed-off-by: Joel Dice <[email protected]> satisfy clippy Signed-off-by: Joel Dice <[email protected]> fix async tests when `component-model-async` enabled Enabling this feature for all tests revealed various missing pieces in the new `concurrent.rs` fiber mechanism, which I've addressed. This adds a bunch of ugly `#[cfg(feature = "component-model-async")]` guards, but those will all go away once I unify the two async fiber implementations. Signed-off-by: Joel Dice <[email protected]> add and modify tests to cover concurrent APIs Primarily, this tests and implements cases where parameters and/or results must be passed via linear memory instead of the stack. Signed-off-by: Joel Dice <[email protected]> `concurrent_{imports|exports}` component macro codegen tests This enables codegen testing of the `concurrent_imports` and `concurrent_exports` options to `wasmtime::component::bindgen` and also fixes code generation for world-level function and resource exports that use the concurrent call style. Signed-off-by: Joel Dice <[email protected]> `concurrent_{imports|exports}` component macro expanded tests This enables testing of the `concurrent_imports` and `concurrent_exports` options in `crates/component-macro/tests/expanded.rs`. Signed-off-by: Joel Dice <[email protected]> add tests/misc_testsuite/component-model-async/*.wast These only test instantiation of components which use various async options and built-ins so far. Next, I'll happy and sad path tests which actually execute code. Signed-off-by: Joel Dice <[email protected]> appease clippy Signed-off-by: Joel Dice <[email protected]> add tests/misc_testsuite/component-model-async/fused.wast Signed-off-by: Joel Dice <[email protected]> add non-panicking bounds checks where appropriate Signed-off-by: Joel Dice <[email protected]> remove post-return bits from async result lift code ...at least until we've determined whether post-return options even make sense for async-lifted exports. Signed-off-by: Joel Dice <[email protected]> fix component-model-async/fused.wast test failure Signed-off-by: Joel Dice <[email protected]> use `enum` types to represent status and event codes Signed-off-by: Joel Dice <[email protected]> fix component-model-async/fused.wast test failure (2nd try) Signed-off-by: Joel Dice <[email protected]> use `gc_types = true` in component-model-async/fused.wast We use `Instruction::RefFunc` when generating adapters for async lifts and/or lowers, which Winch doesn't understand, and apparently `gc_types = true` is what tells the test infra not to use Winch. Signed-off-by: Joel Dice <[email protected]> trap if async function finishes without calling `task.return` Signed-off-by: Joel Dice <[email protected]> update wit-bindgen and fix rebase damage Signed-off-by: Joel Dice <[email protected]> call post-return function if any for async->sync fused calls Signed-off-by: Joel Dice <[email protected]> fix non-component-model-async build; appease clippy Signed-off-by: Joel Dice <[email protected]> bless bindgen output whitespace changes Signed-off-by: Joel Dice <[email protected]> enforce resource borrow requirements for async calls Signed-off-by: Joel Dice <[email protected]> update `wit-bindgen` and simplify `async_borrowing_callee` test Signed-off-by: Joel Dice <[email protected]> call `InstanceFlags::set_may_enter` where appropriate There's still more work to do to fully implement (and test) the reentrance rules for concurrent tasks, but this is a start. Signed-off-by: Joel Dice <[email protected]> finish implementing reentrance checks Signed-off-by: Joel Dice <[email protected]> feat: implement error-context (#1) * feat: initial error-context implementation This commit implements error-context related functions inside the VM, along with tests to ensure that basic error-context.new and error-context.debug-message functionality works. Signed-off-by: Victor Adossi <[email protected]> * wip: add test for error context callee/caller transfer Signed-off-by: Victor Adossi <[email protected]> * wip: test for async context transfer Signed-off-by: Victor Adossi <[email protected]> --------- Signed-off-by: Victor Adossi <[email protected]> run cargo fmt Signed-off-by: Joel Dice <[email protected]> appease clippy Signed-off-by: Joel Dice <[email protected]> pull in Roman's unit stream work; add world-level export test Signed-off-by: Joel Dice <[email protected]> add unit stream tests Signed-off-by: Joel Dice <[email protected]> CI fixes Signed-off-by: Joel Dice <[email protected]> fix non-component-model-async build Signed-off-by: Joel Dice <[email protected]> add round-trip tests with many parameters and results This adds tests to cover round-tripping values whose flattened form exceed `MAX_FLAT_PARAMS`. Although we already had tests to cover this in tests/all/component_model/func.rs, those tests did not involve any flavor of composition; these do. Signed-off-by: Joel Dice <[email protected]> test: add multiple send as initial context stream test Signed-off-by: Victor Adossi <[email protected]> fix Victor's test case Turns out this was the first test case where the host calls a component via an async-lifted export, which in turn calls another component's async-lifted export via a sync-lowered import, which didn't work. Now it does! Signed-off-by: Joel Dice <[email protected]> refactor sync->async component call adapters We no longer use global variables to stash parameters and results due to the hazard of concurrent calls clobbering them. Instead, we now stash them in the `GuestTask` object for the task. Signed-off-by: Joel Dice <[email protected]>

dicej mentioned this pull request Oct 8, 2024

Add support for the Component Model Async ABI #37

Merged

Fix stream.new naming in wasm-tools section

87aa7fd

dicej mentioned this pull request Nov 5, 2024

Add support for async ABI, futures, streams, and errors bytecodealliance/wasm-tools#1895

Merged

4 tasks

dicej mentioned this pull request Nov 7, 2024

Add support for async/streams/futures bytecodealliance/wasmtime#9582

Draft

22 tasks

dicej mentioned this pull request Jan 28, 2025

Support Abi::Array when compiling calls to async intrinsics bytecodealliance/wasmtime#10143

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for the Component Model Async ABI #38

Add support for the Component Model Async ABI #38

dicej commented Oct 8, 2024

Add support for the Component Model Async ABI #38

Are you sure you want to change the base?

Add support for the Component Model Async ABI #38

Conversation

dicej commented Oct 8, 2024