-
Notifications
You must be signed in to change notification settings - Fork 199
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Integrate worlds into the Rust & JS generators #381
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit extracts a few traits and utilities to the `wit-bindgen-core` crate and the `wit-bindgen` CLI. The intention here is to make it simple to implement this same generation scheme for other languages, such as Python, and additionally provide a path forward to centering the CLI interface around world files.
This is intended to be the first step down the path of reorienting the Wasmtime generator around worlds instead of single interfaces through the new `WorldGenerator` trait rather than the old `Generator` trait.
pchickey
approved these changes
Oct 18, 2022
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a big enough diff that I can't verify every line of it, but as we discussed, the code transformation that happened here is in the correct direction, and I trust that since the tests are passing that this is correct.
alexcrichton
added a commit
to alexcrichton/witx-bindgen
that referenced
this pull request
Oct 20, 2022
This commit is the equivalent of bytecodealliance#381 but for Wasmtime Python host bindings. This is a large-scale refactor of the internals of the Python host generator in which I took quite a few liberties in internal restructuring as well as output restructuring. The tests are probably the best to review and better reflect what changed, but there are some aspects of worlds that the tests are not currently exercising which we'll want to add with the introduction of worlds in the future. This means that all host generators are now working with worlds as input rather than individual `*.wit` files, and the only two remaining generators are the C and Java generators for guests (which I hope are easier). The high level summary of the new output is: out_dir/ __init__.py # top-level component exports/definitions types.py # shared type information, currently just `Result` imports/ __init__.py # only here if something is imported foo.py # type and protocol definition per interface exports/ __init__.py # only here if an instance is exported bar.py # one per exported instance "Default exports" will show up on the generated structure in `out_dir/__init__.py` so all runtime tests, for example, do not generate `exports` at this time. Lots of fiddly stuff went into structuring this all right to get past `mypy` and additionally try to avoid name conflicts. It's still somewhat easy to have name conflicts but ideally they're of the more esoteric category rather than "really easy to run into".
alexcrichton
added a commit
that referenced
this pull request
Oct 24, 2022
* Reimplement the host Python generator with worlds This commit is the equivalent of #381 but for Wasmtime Python host bindings. This is a large-scale refactor of the internals of the Python host generator in which I took quite a few liberties in internal restructuring as well as output restructuring. The tests are probably the best to review and better reflect what changed, but there are some aspects of worlds that the tests are not currently exercising which we'll want to add with the introduction of worlds in the future. This means that all host generators are now working with worlds as input rather than individual `*.wit` files, and the only two remaining generators are the C and Java generators for guests (which I hope are easier). The high level summary of the new output is: out_dir/ __init__.py # top-level component exports/definitions types.py # shared type information, currently just `Result` imports/ __init__.py # only here if something is imported foo.py # type and protocol definition per interface exports/ __init__.py # only here if an instance is exported bar.py # one per exported instance "Default exports" will show up on the generated structure in `out_dir/__init__.py` so all runtime tests, for example, do not generate `exports` at this time. Lots of fiddly stuff went into structuring this all right to get past `mypy` and additionally try to avoid name conflicts. It's still somewhat easy to have name conflicts but ideally they're of the more esoteric category rather than "really easy to run into". * Try to fix windows
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The initial goal of this PR was to generalize the work in the JS generator to enable migrating the Python generator to the same infrastructure. In doing so, however, it created a trait to process a world as defined today: a list of imports, a list of exports, and an optional default interface. I wanted to prove out this trait and its usefulness and so I opted to migrate the Rust guest and host generators to this trait as well.
This ended up being much larger than intended, but all of this I believe is inevitably needed for #314 or worlds in general. To be clear worlds do not yet exist in wit-bindgen but this PR aims to create a "hole" in
wit-bindgen
where a world would clearly fill. To that end the various changes implemented in this PR are:Boilerplate code in the JS code generator was reduced and moved to the
wit-bindgen-core
infrastructure. The goal here was to remove the need to hardcode in the JS code generator what will also be required in the Python code generator. This created theComponentGenerator
trait which is specifically tailored for instantiating a single component.The new
ComponentGenerator
also inherits from a newWorldGenerator
trait. This is designed to be the new defacto interface for all code generators. There's still some existing ones using "legacy worlds" which is the oldGenerator
trait but my plan is to eventually delete that once all generators have been migrated. TheWorldGenerator
trait takes aComponentInterfaces
argument which is the current implementation understanding of a world.The guest Rust generator was updated to the new
WorldGenerator
trait. Similarly the host Wasmtime generator was updated to this new trait. This involved quite a bit of code movement but it also ended up filling out some novel new features. For example it's now more integrated to create exported instances from Rust guests or possible to consume these instances from Rust hosts. Overall the end-to-end representation of a world should be more filled out now.The guest Rust generator's way of modeling exports has changed to use
standalone
by default. I've removed thestandalone
option here and opted to always have a macro which generates theno_mangle
functions.Lots of fiddly bits around the Wasmtime and the Rust guest generators have changed. Everything prior should still be possible and with even more now that the full expressiveness of our current world representation is supported. It's probably best to review the test changes here to see how things work.
I'd also like to specifically call out that the old Rust
import!
andexport!
macros are now removed. There's now only onegenerate!
macro. Eventually this is intended to begenerate!("my-world.wit")
but we don't have a parser for worlds right now so the next-best thing is that you're required to dogenerate!({ ... })
with manualimport
,export
, anddefault
options. This is not intended to be ergonomic or even easily understandable at this time, it's just intended to work and last through the transition to worlds.Overall to reiterate, the goal of this PR is to create a round hole for the round peg of worlds to slot into. There's a number of locations where ergonomics and easy-to-understandability are left on the table because a single concept of worlds in
*.wit
files should solve it. Furthermore this does not address issues such as sharing types across interfaces, that's left for future refactorings.