Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate worlds into the Rust & JS generators #381

Merged
merged 6 commits into from
Oct 18, 2022

Conversation

alexcrichton
Copy link
Member

The initial goal of this PR was to generalize the work in the JS generator to enable migrating the Python generator to the same infrastructure. In doing so, however, it created a trait to process a world as defined today: a list of imports, a list of exports, and an optional default interface. I wanted to prove out this trait and its usefulness and so I opted to migrate the Rust guest and host generators to this trait as well.

This ended up being much larger than intended, but all of this I believe is inevitably needed for #314 or worlds in general. To be clear worlds do not yet exist in wit-bindgen but this PR aims to create a "hole" in wit-bindgen where a world would clearly fill. To that end the various changes implemented in this PR are:

  • Boilerplate code in the JS code generator was reduced and moved to the wit-bindgen-core infrastructure. The goal here was to remove the need to hardcode in the JS code generator what will also be required in the Python code generator. This created the ComponentGenerator trait which is specifically tailored for instantiating a single component.

  • The new ComponentGenerator also inherits from a new WorldGenerator trait. This is designed to be the new defacto interface for all code generators. There's still some existing ones using "legacy worlds" which is the old Generator trait but my plan is to eventually delete that once all generators have been migrated. The WorldGenerator trait takes a ComponentInterfaces argument which is the current implementation understanding of a world.

  • The guest Rust generator was updated to the new WorldGenerator trait. Similarly the host Wasmtime generator was updated to this new trait. This involved quite a bit of code movement but it also ended up filling out some novel new features. For example it's now more integrated to create exported instances from Rust guests or possible to consume these instances from Rust hosts. Overall the end-to-end representation of a world should be more filled out now.

  • The guest Rust generator's way of modeling exports has changed to use standalone by default. I've removed the standalone option here and opted to always have a macro which generates the no_mangle functions.

Lots of fiddly bits around the Wasmtime and the Rust guest generators have changed. Everything prior should still be possible and with even more now that the full expressiveness of our current world representation is supported. It's probably best to review the test changes here to see how things work.

I'd also like to specifically call out that the old Rust import! and export! macros are now removed. There's now only one generate! macro. Eventually this is intended to be generate!("my-world.wit") but we don't have a parser for worlds right now so the next-best thing is that you're required to do generate!({ ... }) with manual import, export, and default options. This is not intended to be ergonomic or even easily understandable at this time, it's just intended to work and last through the transition to worlds.

Overall to reiterate, the goal of this PR is to create a round hole for the round peg of worlds to slot into. There's a number of locations where ergonomics and easy-to-understandability are left on the table because a single concept of worlds in *.wit files should solve it. Furthermore this does not address issues such as sharing types across interfaces, that's left for future refactorings.

This commit extracts a few traits and utilities to the
`wit-bindgen-core` crate and the `wit-bindgen` CLI. The intention here
is to make it simple to implement this same generation scheme for other
languages, such as Python, and additionally provide a path forward to
centering the CLI interface around world files.
This is intended to be the first step down the path of reorienting
the Wasmtime generator around worlds instead of single interfaces
through the new `WorldGenerator` trait rather than the old `Generator`
trait.
@alexcrichton alexcrichton requested a review from pchickey October 18, 2022 20:01
Copy link
Contributor

@pchickey pchickey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a big enough diff that I can't verify every line of it, but as we discussed, the code transformation that happened here is in the correct direction, and I trust that since the tests are passing that this is correct.

@alexcrichton alexcrichton merged commit 614e401 into bytecodealliance:main Oct 18, 2022
@alexcrichton alexcrichton deleted the use-worlds branch October 18, 2022 21:29
alexcrichton added a commit to alexcrichton/witx-bindgen that referenced this pull request Oct 20, 2022
This commit is the equivalent of bytecodealliance#381 but for Wasmtime Python host
bindings. This is a large-scale refactor of the internals of the Python
host generator in which I took quite a few liberties in internal
restructuring as well as output restructuring. The tests are probably
the best to review and better reflect what changed, but there are some
aspects of worlds that the tests are not currently exercising which
we'll want to add with the introduction of worlds in the future.

This means that all host generators are now working with worlds as input
rather than individual `*.wit` files, and the only two remaining
generators are the C and Java generators for guests (which I hope are
easier).

The high level summary of the new output is:

    out_dir/
      __init__.py             # top-level component exports/definitions
      types.py                # shared type information, currently just `Result`
      imports/
        __init__.py           # only here if something is imported
        foo.py                # type and protocol definition per interface
      exports/
        __init__.py           # only here if an instance is exported
        bar.py                # one per exported instance

"Default exports" will show up on the generated structure in
`out_dir/__init__.py` so all runtime tests, for example, do not generate
`exports` at this time. Lots of fiddly stuff went into structuring this
all right to get past `mypy` and additionally try to avoid name
conflicts. It's still somewhat easy to have name conflicts but ideally
they're of the more esoteric category rather than "really easy to run
into".
alexcrichton added a commit that referenced this pull request Oct 24, 2022
* Reimplement the host Python generator with worlds

This commit is the equivalent of #381 but for Wasmtime Python host
bindings. This is a large-scale refactor of the internals of the Python
host generator in which I took quite a few liberties in internal
restructuring as well as output restructuring. The tests are probably
the best to review and better reflect what changed, but there are some
aspects of worlds that the tests are not currently exercising which
we'll want to add with the introduction of worlds in the future.

This means that all host generators are now working with worlds as input
rather than individual `*.wit` files, and the only two remaining
generators are the C and Java generators for guests (which I hope are
easier).

The high level summary of the new output is:

    out_dir/
      __init__.py             # top-level component exports/definitions
      types.py                # shared type information, currently just `Result`
      imports/
        __init__.py           # only here if something is imported
        foo.py                # type and protocol definition per interface
      exports/
        __init__.py           # only here if an instance is exported
        bar.py                # one per exported instance

"Default exports" will show up on the generated structure in
`out_dir/__init__.py` so all runtime tests, for example, do not generate
`exports` at this time. Lots of fiddly stuff went into structuring this
all right to get past `mypy` and additionally try to avoid name
conflicts. It's still somewhat easy to have name conflicts but ideally
they're of the more esoteric category rather than "really easy to run
into".

* Try to fix windows
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants