Automatic binding generation #264

madsmtm · 2022-09-08T12:10:52Z

This is very similar to using bindgen, but I went with creating things from scratch because:

bindgen is kinda intimidating, knowing so little about libclang, so I started here first; I'd still like to upstream some of this, but I'll start here for now.
This only has to run on my machine every once in a while, whereas bindgen has to handle all C code under the sun. A few manual fixes and such are acceptable.
I'd like to run this multiple times for different targets (iOS, macOS, tvOS, ... - GNUStep will probably suffer a bit here), and then merge the result using #[cfg] so that we can ship one, ready-to-use library.
I'd like to enrich the output using separate files (.apinotes or similar), and such a thing will probably never belong in bindgen
Other things that are harder to integrate into something existing, for example availability attributes.

That said, I have taken inspiration from the existing Objective-C implementation in bindgen, credits to @simlay.

In the end, I'm envisioning something like this (so objc2 would be the Apple equivalent to the windows crate):

objc2/src/
  foundation/
    generated/
      ...
    unsafe_fns.toml // Or `Foundation.apinotes`, or similar
    string.rs
    mutable_string.rs
    ...
  appkit/...
  core_data/...
  core_display/...
  // And so on, for the Apple frameworks that expose an Objective-C interface

See #85 (comment) for a bit of details on what Swift does.

madsmtm · 2022-09-09T17:50:35Z

I estimated the size of the code if we included all of Apple's frameworks (assuming Rust and C have similar line ratios); it would be around 13MB zipped, 50MB unzipped. For reference, the windows crate is ~14MB zipped, ~200MB unzipped.

madsmtm · 2022-09-09T18:30:02Z

Related projects:

madsmtm · 2022-09-19T11:42:33Z

I probably won't put this under objc2, instead I'll create a new crate with the frameworks. Possible names, in order of my current preference:

apple, is an empty crate owned by @zu1kd, have sent them an email to discuss name transfer
icrate, a play on Apple's "i"-prefixing (iPhone/iPad/...)
ilib, similar to above
frameworks, ambiguous since also commonly used for "web frameworks"
appel, wordplay

madsmtm · 2022-09-23T22:59:52Z

Example of how I want at least some of AppKit to look like: https://github.com/rust-windowing/winit/tree/fafdedfb7d3a7370ca4b01108f7713b685633164/src/platform_impl/macos/appkit

madsmtm · 2022-10-04T21:06:56Z

init methods always return the same type as the receiver: https://clang.llvm.org/docs/AutomaticReferenceCounting.html#related-result-types

How can we model this nicely?

madsmtm · 2022-10-05T18:46:11Z

Idea: use Swifty names? https://github.com/apple/swift-evolution/blob/main/proposals/0005-objective-c-name-translation.md

madsmtm · 2022-10-05T18:47:00Z

Idea: Use (parts of) Swift's test suite (for example this one for errors: https://github.com/apple/swift/blob/7123d2614b5f222d03b3762cb110d27a9dd98e24/test/Inputs/clang-importer-sdk/usr/include/errors.h)

madsmtm · 2022-10-10T18:07:03Z

Clang has "module maps" for figuring out what to import from Objective-C headers - we should use those!

ericmarkmartin · 2022-10-30T19:08:37Z

Hey,

I'm really excited about this enhancement and would find it super useful for some work I'm doing on binding the AuthenticationServices framework. That in mind, I have a couple of high-level questions.

do you have a rough ETA for when this thing might land? If it's soon-ish I'll probably hold off on writing my own bindings but otherwise I might just proceed on my own.
if you think this is still a little ways out, is there any way I could help to bring this around sooner?

madsmtm · 2022-10-31T19:19:28Z

Currently I'm trying to push forwards on an initial version that actually compiles Foundation and AppKit.

Primary missing items are:

Proper imports
Struct definitions
A few remaining typedefs
Allow returning Id from extern_methods! #244
The remaining stuff to get things to compile

Timeline on this is unknown, but I have quite a lot of free time this week, so I should be able to get pretty far. I can probably give you a better estimate at the end of the week, if I'm not done by then.

Once I have this finished, there'll be lots of smaller things to work on, which I could probably use some help with, including:

Actual type-safety (creating newtypes instead of type-aliases for enums, typedefs and so on, allowing us to later on mark various methods as safe)
Extern functions
Inline functions (requires quite a lot, since we effectively have to translate C code to Rust. But we can probably get quite far with a few tricks)
Block support
Protocols: Add extern_protocol! macro and ProtocolType trait #250
"out" parameters: Handling objects passed as "out parameters" #277
Autogenerated documentation (using code comments, and ideally data from developer.apple.com)
Possibly a more "swifty" translation (tweaking method names, making enums associated constants). Probably better to do after we have autogenerated documentation, since there is great value in the current design where you can very easily find what you're looking for

I'll try to see if I can figure out a good way for you / somebody else to contribute with some of that, when I get there.

madsmtm · 2022-11-01T14:37:43Z

I could use some guidance though: I'll admit I'm not the best at git, and I'm unsure if storing the generated result in git is the best solution? It does make audits (security and just general) easier, at the cost of blowing up the amount files we store. An alternative could also be to create a new repo just for it.

But what do you think?

simlay

I could use some guidance though: I'll admit I'm not the best at git, and I'm unsure if storing the generated result in git is the best solution? It does make audits (security and just general) easier, at the cost of blowing up the amount files we store. An alternative could also be to create a new repo just for it.

But what do you think?

Putting it in a different repo isn't a bad idea but I think in general what I'd say is that the generated stuff in tracking should be either for some kind of sys-crate (or icrate) or for integration tests. Bindgen keeps the input headers and the expectations in git and verifies that they match as expected in CI and the output does compile. Using the full frameworks to test is useful to know breadth of functionality but I believe it can change from macOS/SDK version to version.

After reviewing some stuff, it seems like the header-translator crate is meant to run as a binary and then either store the output in the crate and tracking. I tend to like using a generator crate in a build script.

header-translator/src/main.rs

simlay · 2022-11-01T15:45:28Z

header-translator/src/main.rs

+    });
+}
+
+fn output_files(


Should you want to use this crate as a build script, I think output_files (and other stuff) will need to go in the lib.rs

madsmtm · 2022-11-01T20:19:14Z

I tend to like using a generator crate in a build script

Yeah, I'm aware of that approach, my desire with header-translator/icrate was to do things a bit differently than bindgen, to improve:

Compilation speed, since the work is only done once on my local machine, and not on every user's machine (here I'm hoping that download speed will not be a limiting factor)
Documentation, since items that are only available on one platform would be hidden behind appropriate cfgs
The need to have a specific SDK version installed (would help with cross-compiling on Linux)

I believe it can change from macOS/SDK version to version.

I think my desire was to manually keep icrate up to date with new SDK versions. I'm currently compiling for the macOS 12.3 SDK, on a machine running macOS 10.14.6.

We can't parse it yet though, see #250

notgull · 2022-12-16T23:47:02Z

Well, I was writing an automatic binding generator for AppKit, only to find out that you're doing it already; and from a glance it looks like your version is way ahead of mine! :-)

My only complaint would be that, in my opinion, having the setup use Deref in order to emulate inheritance isn't hygenic. If it were up to me, I'd do something like what gtk-rs does:

// Have this trait.
pub trait IsA<T> {}

pub struct NSDate { /* .. */ }
impl IsA<NSDate> for NSDate {}
impl IsA<NSObject> for NSObject {}

// Extension trait.
pub trait NSDateExt : IsA<NSDate> { /* ... */ }
impl<T: IsA<NSDate>> NSDateExt for T {}

That being said, I'm very excited to see this once it reaches readiness. Please let me know if there's anything I can do for this.

See full history in 1c4c875. Add initial header translation Closer to usable Mostly works with NSCursor Mostly works with NSAlert.h Refactor a bit AppKit is now parse-able handle reserved keywords Handle protocols somewhat Handle the few remaining entity kinds Works with Foundation Cleanup Refactor Refactor Method to (almost) be PartialEq Parse more things Parse NSConsumed Verify memory management More work Fix reserved keywords Refactor statements Add initial availability Prepare RustType Split RustType up in parse and ToToken part Add icrate Add config files Temporarily disable protocol generation Generate files Add initial generated files for Foundation Skip "index" header Add basic imports Allow skipping methods Properly emit `unsafe` Make classes public Rename objc feature flag Improve imports somewhat Further import improvements Handle basic typedefs Improve generics handling Improve pointers to objects Refactor RustType::TypeDef Mostly support generics Refactor config setup Small fixes Support nested generics Comment out a bit of debug logging Emit all files Parse sized integer types Parse typedefs that map to other typedefs Appease clippy Add `const`-parsing for RustType::Id Parse Objective-C in/out/inout/bycopy/byref/oneway qualifiers Fix `id` being emitted when it actually specifies a protocol Make AppKit work again Parse all qualifiers, in particular lifetime qualifiers More consistent ObjCObjectPointer parsing Validate some lifetime attributes Fix out parameters (except for NSError) Assuming we find a good solution to #277 Refactor Stmt objc declaration parsing Clean up how return types work Refactor property parsing Fixes their order to be the same as in the source file Add support for functions taking NSError as an out parameter Assuming we do #276 Change icrate directory layout Refactor slightly Refactor file handling to allow for multiple frameworks simultaneously Put method output inside an extern_methods! call We'll want this no matter what, since it'll allow us to extend things with availability attributes in the future. Use extern_methods! functionality To cut down on the amount of code, which should make things easier to review and understand. This uses features which are not actually yet done, see #244. Not happy with the formatting either, but not sure how to fix that? Manually fix the formatting of attribute macros in extern_methods! Add AppKit bindings Speed things up by optionally formatting at the end instead Prepare for parsing more than one SDK Specify a minimum deployment target Document SDK situation Parse headers on iOS as well Refactor stmt parsing a bit Remove Stmt::FileImport and Stmt::ItemImport These are not nearly enough to make imports work well anyhow, so I'll rip it out and find a better solution Do preprocessing step explicitly as the first thing Refactor so that file writing is done using plain Display Allows us to vastly improve the speed, as well as allowing us to make the output much prettier wrt. newlines and such in the future (proc_macro2 / quote output is not really meant to be consumed by human eyes) Improve whitespace in generated files and add warning header Don't crash on unions Add initial enum parsing Add initial enum expr parsing Add very simple enum output Fix duplicate enum check Improve enum expr parsing This should make it easier for things to work on 32-bit platforms Add static variable parsing Add a bit of WIP code Add function parsing Fix generic struct generation Make &Class as return type static Trim unnecessary parentheses Fix generics default parameter Remove methods that are both instance and class methods For now, until we can solve this more generally Skip protocols that are also classes Improve imports setups Bump recursion limit Add MacTypes.h type translation Fix int64_t type translation Make statics public Fix init methods Make __inner_extern_class allowing trailing comma in generics Attempt to improve Rust's parsing speed of icrate Custom NSObject TMP import Remove NSProxy Temporarily remove out parameter setup Add struct support Add partial support for "related result types" Refactor typedef parsing a bit Output remaining typedefs Fix Option<Sel> and *mut bool Fix almost all remaining type errors in Foundation Skip statics whoose value we cannot find Fix anonymous enum types Fix AppKit duplicate methods Add CoreData Properly fix imports Add `abstract` keyword Put enum and static declarations behind a macro Add proper IncompleteArray parsing Refactor type parsing Make NSError** handling happen in all the places that it does with Swift Refactor Ty a bit more Make Display for RustType always sound Add support for function pointers Add support for block pointers Add extern functions Emit protocol information We can't parse it yet though, see #250 Make CoreData compile Make AppKit compile Add support for the AuthenticationServices framework Do clang < v13 workarounds without modifying sources Refactor Foundation fixes

madsmtm · 2022-12-21T19:34:23Z

My only complaint would be that, in my opinion, having the setup use Deref in order to emulate inheritance isn't hygenic. If it were up to me, I'd do something like what gtk-rs does:
// Have this trait.
pub trait IsA<T> {}

pub struct NSDate { /* .. */ }
impl IsA<NSDate> for NSDate {}
impl IsA<NSObject> for NSObject {}

// Extension trait.
pub trait NSDateExt : IsA<NSDate> { /* ... */ }
impl<T: IsA<NSDate>> NSDateExt for T {}

Yeah, still very much experimenting with what's the best approach!

I'm not sure what you mean by "isn't hygenic", would you care to elaborate?

That being said, I'm very excited to see this once it reaches readiness. Please let me know if there's anything I can do for this.

Sure, there will be fairly soon.

I'm done with the rebase and do actually have something workable over in #308 now, once that is merged it should become easier to collaborate.

See full history in 1c4c875. Add initial header translation Closer to usable Mostly works with NSCursor Mostly works with NSAlert.h Refactor a bit AppKit is now parse-able handle reserved keywords Handle protocols somewhat Handle the few remaining entity kinds Works with Foundation Cleanup Refactor Refactor Method to (almost) be PartialEq Parse more things Parse NSConsumed Verify memory management More work Fix reserved keywords Refactor statements Add initial availability Prepare RustType Split RustType up in parse and ToToken part Add icrate Add config files Temporarily disable protocol generation Generate files Add initial generated files for Foundation Skip "index" header Add basic imports Allow skipping methods Properly emit `unsafe` Make classes public Rename objc feature flag Improve imports somewhat Further import improvements Handle basic typedefs Improve generics handling Improve pointers to objects Refactor RustType::TypeDef Mostly support generics Refactor config setup Small fixes Support nested generics Comment out a bit of debug logging Emit all files Parse sized integer types Parse typedefs that map to other typedefs Appease clippy Add `const`-parsing for RustType::Id Parse Objective-C in/out/inout/bycopy/byref/oneway qualifiers Fix `id` being emitted when it actually specifies a protocol Make AppKit work again Parse all qualifiers, in particular lifetime qualifiers More consistent ObjCObjectPointer parsing Validate some lifetime attributes Fix out parameters (except for NSError) Assuming we find a good solution to #277 Refactor Stmt objc declaration parsing Clean up how return types work Refactor property parsing Fixes their order to be the same as in the source file Add support for functions taking NSError as an out parameter Assuming we do #276 Change icrate directory layout Refactor slightly Refactor file handling to allow for multiple frameworks simultaneously Put method output inside an extern_methods! call We'll want this no matter what, since it'll allow us to extend things with availability attributes in the future. Use extern_methods! functionality To cut down on the amount of code, which should make things easier to review and understand. This uses features which are not actually yet done, see #244. Not happy with the formatting either, but not sure how to fix that? Manually fix the formatting of attribute macros in extern_methods! Add AppKit bindings Speed things up by optionally formatting at the end instead Prepare for parsing more than one SDK Specify a minimum deployment target Document SDK situation Parse headers on iOS as well Refactor stmt parsing a bit Remove Stmt::FileImport and Stmt::ItemImport These are not nearly enough to make imports work well anyhow, so I'll rip it out and find a better solution Do preprocessing step explicitly as the first thing Refactor so that file writing is done using plain Display Allows us to vastly improve the speed, as well as allowing us to make the output much prettier wrt. newlines and such in the future (proc_macro2 / quote output is not really meant to be consumed by human eyes) Improve whitespace in generated files and add warning header Don't crash on unions Add initial enum parsing Add initial enum expr parsing Add very simple enum output Fix duplicate enum check Improve enum expr parsing This should make it easier for things to work on 32-bit platforms Add static variable parsing Add a bit of WIP code Add function parsing Fix generic struct generation Make &Class as return type static Trim unnecessary parentheses Fix generics default parameter Remove methods that are both instance and class methods For now, until we can solve this more generally Skip protocols that are also classes Improve imports setups Bump recursion limit Add MacTypes.h type translation Fix int64_t type translation Make statics public Fix init methods Make __inner_extern_class allowing trailing comma in generics Attempt to improve Rust's parsing speed of icrate Custom NSObject TMP import Remove NSProxy Temporarily remove out parameter setup Add struct support Add partial support for "related result types" Refactor typedef parsing a bit Output remaining typedefs Fix Option<Sel> and *mut bool Fix almost all remaining type errors in Foundation Skip statics whoose value we cannot find Fix anonymous enum types Fix AppKit duplicate methods Add CoreData Properly fix imports Add `abstract` keyword Put enum and static declarations behind a macro Add proper IncompleteArray parsing Refactor type parsing Make NSError** handling happen in all the places that it does with Swift Refactor Ty a bit more Make Display for RustType always sound Add support for function pointers Add support for block pointers Add extern functions Emit protocol information We can't parse it yet though, see #250 Make CoreData compile Make AppKit compile Add support for the AuthenticationServices framework Do clang < v13 workarounds without modifying sources Refactor Foundation fixes

madsmtm · 2022-12-23T14:59:35Z

A possible source of at least a bit of docs is also https://github.com/xamarin/apple-api-docs.

In general, Xamarin has done a lot of work in this space, I should try to take a look at their stuff at some point

madsmtm · 2022-12-23T17:58:25Z

#308 is done 🎉

Will try to release a new version of icrate to crates.io today, so that you can play with it. I've opened tagged issues for some of the remaining stuff - do free to comment on any of them if you have suggestions for how to do something differently (especially ergonomics-wise, the hardest part in all of this is in the design work ;) )

madsmtm added the enhancement New feature or request label Sep 8, 2022

This was referenced Sep 8, 2022

bindgen support #85

Closed

The deal with mutability #265

Closed

madsmtm force-pushed the header-translator branch from bc10d5a to c6ef627 Compare September 9, 2022 07:55

madsmtm mentioned this pull request Sep 17, 2022

Move to objc2 rust-windowing/glutin#1461

Merged

4 tasks

madsmtm mentioned this pull request Sep 20, 2022

Using metal-cpp for auto-generating bindings gfx-rs/metal-rs#220

Open

madsmtm force-pushed the header-translator branch from 075c18c to 167ad93 Compare September 23, 2022 14:43

madsmtm mentioned this pull request Sep 23, 2022

Replace objc with objc2? ryanmcgrath/cacao#28

Open

madsmtm mentioned this pull request Oct 5, 2022

Better error handling in msg_send! and msg_send_id! #276

Merged

madsmtm force-pushed the header-translator branch 4 times, most recently from dedf7d4 to 420f9a0 Compare October 9, 2022 01:28

This was referenced Oct 26, 2022

How do you not cause memory leaks? SSheldon/rust-objc#117

Closed

async support #279

Open

madsmtm force-pushed the header-translator branch from e0ff895 to a8d2ada Compare October 31, 2022 16:10

simlay reviewed Nov 1, 2022

View reviewed changes

madsmtm force-pushed the header-translator branch from a8d2ada to d6eefa8 Compare November 1, 2022 20:16

madsmtm added 14 commits December 8, 2022 11:47

Put enum and static declarations behind a macro

f495d75

Add proper IncompleteArray parsing

98d8dd8

Refactor type parsing

4e692fe

Make NSError** handling happen in all the places that it does with Swift

b2815b4

Refactor Ty a bit more

7f61929

Make Display for RustType always sound

78fcdd1

Add support for function pointers

d78b975

Add support for block pointers

2d25da2

Add extern functions

0bcf7d0

Emit protocol information

35988ba

We can't parse it yet though, see #250

Make CoreData compile

2080c5e

Make AppKit compile

baff198

Add support for the AuthenticationServices framework

cb00d95

Do clang < v13 workarounds without modifying sources

d1ae521

madsmtm force-pushed the header-translator branch from 56827ff to d1ae521 Compare December 8, 2022 10:58

Refactor Foundation fixes

1c4c875

madsmtm mentioned this pull request Dec 21, 2022

Can't have NSString and NSNumber as values in an NSDictionary #306

Closed

madsmtm mentioned this pull request Dec 21, 2022

Automatic binding generation, second PR #308

Merged

This was referenced Dec 23, 2022

Improve objc2::foundation #67

Closed

Autogenerate documentation in framework crates #309

Open

madsmtm added the A-framework Affects the framework crates and the translator for them label Dec 23, 2022

madsmtm closed this Dec 23, 2022

madsmtm deleted the header-translator branch December 23, 2022 17:59

madsmtm mentioned this pull request Dec 24, 2022

Improve compilation time of icrate #311

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automatic binding generation #264

Automatic binding generation #264

madsmtm commented Sep 8, 2022 •

edited

Loading

madsmtm commented Sep 9, 2022 •

edited

Loading

madsmtm commented Sep 9, 2022 •

edited

Loading

madsmtm commented Sep 19, 2022

madsmtm commented Sep 23, 2022

madsmtm commented Oct 4, 2022

madsmtm commented Oct 5, 2022

madsmtm commented Oct 5, 2022

madsmtm commented Oct 10, 2022

ericmarkmartin commented Oct 30, 2022

madsmtm commented Oct 31, 2022

madsmtm commented Nov 1, 2022

simlay left a comment

simlay Nov 1, 2022

madsmtm commented Nov 1, 2022

notgull commented Dec 16, 2022

madsmtm commented Dec 21, 2022

madsmtm commented Dec 23, 2022

madsmtm commented Dec 23, 2022

Automatic binding generation #264

Automatic binding generation #264

Conversation

madsmtm commented Sep 8, 2022 • edited Loading

madsmtm commented Sep 9, 2022 • edited Loading

madsmtm commented Sep 9, 2022 • edited Loading

madsmtm commented Sep 19, 2022

madsmtm commented Sep 23, 2022

madsmtm commented Oct 4, 2022

madsmtm commented Oct 5, 2022

madsmtm commented Oct 5, 2022

madsmtm commented Oct 10, 2022

ericmarkmartin commented Oct 30, 2022

madsmtm commented Oct 31, 2022

madsmtm commented Nov 1, 2022

simlay left a comment

Choose a reason for hiding this comment

simlay Nov 1, 2022

Choose a reason for hiding this comment

madsmtm commented Nov 1, 2022

notgull commented Dec 16, 2022

madsmtm commented Dec 21, 2022

madsmtm commented Dec 23, 2022

madsmtm commented Dec 23, 2022

madsmtm commented Sep 8, 2022 •

edited

Loading

madsmtm commented Sep 9, 2022 •

edited

Loading

madsmtm commented Sep 9, 2022 •

edited

Loading