UniFFiing types and metadata: splitting out UDL

[mhammond]

If agreed, the outcome of this issue will be the following:

• Split the UDL parsing code into its own crate, having it emit uniffi_meta structs
• Have uniffi_bindgen depend only on uniffi_meta - ie, it knows nothing about weedle, udl, etc
• uniffi_bindgen however does remain responsible for generating "scaffolding" (ie, the rust code to support what's defined in the UDL)

The end result is that:

• UDL is "demoted" (ie, no longer a first-class citizen of uniffi_bindgen)

• uniffi_macros is "promoted" to the canonical source for uniffi_bindgen.

• uniffi_bindgen continues to use its "interface" objects - eg, all bindings continue to use a uniffi_bindgen::interface::Object, it just that these Objects are always instantiated from uniffi_meta (whereas now they are constructed via both meta and weedle types)

• A downside of this is that uniffi_meta will need to grow to accomodate metadata specific only to bindings - eg, the boolean to indicate whether an argument is ByRef will be added to uniffi_meta even though bindings and the procmacros never need it.

• uniffi_meta will similarly need to grow support for everything binding specific that can be expressed in UDL, even if uniffi_macros can't yet do that.

I propose:

Step 1

• Move core type, such as Type, Literal and FfiType, into uniffi_meta, removing the existing equivalent types.
  ** Even though the uniffi_meta versions of these structs are arguably cleaner, we stick with the OG type struct to avoid breaking bindings.

Step 2

• A new crate, uniffi_udl: All UDL specific code in uniffi_bindgen is moved to this crate.
• uniffi_udl depends on uniffi_meta, but not uniffi_bindgen - thus it deals exclusively with uniffi_meta.
• uniffi_meta grows support for fields that are not yet supported by uniffi_macros but which uniffi_bindgen needs.
• uniffi_bindgen then springs a ComponentInterface and TypeUniverse into existence only via uniffi_meta types.

Steps N++

• See if there's further scope to have the scaffolding and binding generation become more specialised.

@jplatte @bendk any thoughts or objections?

┆Issue is synchronized with this Jira Task
┆Issue Number: UNIFFI-285

[jplatte]

What's the problem with the current structure? Why move everything around now when we'll have to move everything around or do other big refactorings again after UDL is removed? Do you expect UDL to stay around for long once the proc-macros have feature parity?

[mhammond]

I personally doubt UDL will be removed for a number of years - I'm certainly not going to force our internal projects which are quite stable to either migrate away from UDL or stay on old versions of UniFFI. I even suspect some people will choose to use UDL even after proc-macros have feature parity, certainly until we have nice tooling for generating docs.

IMO effective tooling for the consumers will be critical - the consumers of our crates are not Rust engineers, and currently the UDL doubles as user-documentation. I don't think asking them to grep Rust code for macros to try and reverse engineer the foreign interface they use is going to be warmly received. Casual conversations with these engineers imply it might never be well received - the use of UDL is a feature to some of our users, not a bug.

[bendk]

I like this overall. Even if we decide to kill UDL in the next year or so, this could be seen as a good first step.

uniffi_bindgen however does remain responsible for generating "scaffolding" (ie, the rust code to support what's defined in the UDL)

I wonder if we could move the scaffolding generation to another crate or maybe even to a module inside uniffi_macros behind a feature flag. This seems like it simplifies the overall pipeline some.

Ideally, this code would parse the UDL file, generate a series of macros calls, and then go through more-or-less the same codepath. This makes the testing a lot simpler. I had to duplicate the ext-types library in #1600 because we need to test both the proc-macro and UDL-based code.

Maybe this is part of the N++ plan, I'm not sure how to best sequence all of the work here.

[mhammond]

FWIW, A preview of "step 2" is at https://github.com/mhammond/uniffi-rs/pull/5/files. It's quite a large patch, but I quite like the outcome.