Supporting GameActivity from Android Game Developer Kit (and other Activity subclasses)

[rib]

I was wondering if anyone here has investigated building Rust apps base on Google's newer GameActivity which looks like it can be viewed as a more feature-full NativeActivity which provides more C/C++ native bindings / glue for things like input/IME and controllers.

Ref: https://android-developers.googleblog.com/2021/07/introducing-android-game-development-kit.html

GameActivity is also notably based on AppCompatActivity that itself should offer improved compatibility across different versions of Android.

Also ref this overview blog post: https://medium.com/kayvan-kaseb/using-c-c-libraries-for-android-game-development-63fad09e0711

On a related note I'm currently unsure how heavily tied android-ndk-rs is to NativeActivity.

Right now I'm writing a Rust library that interacts with Bluetooth on Android where I need to have my own subclass of Activity for being able to use Activity::startIntentSenderForResult and override onActivityResult. (The way Android's Bluetooth Companion API works is via an Intent that sends back a result when the user chooses a device)

Right now I'm testing my library with a simple test application but I was also hoping to be able to build a Bevy or egui application using winit based on a subclass of GameActivity.

Starting to poke around winit and seeing how it depends on android-ndk-rs I'm a bit unsure how heavily these things are currently tied to NativeActivity and also tools like cargo ndk/cargo apk (I'm currently using rust-android-gradle )

Sorry the issue conflates a few questions. Besides raising the idea as a kind of feature request, I'm also hoping someone familiar with android-ndk-rs might be able to highlight any particular issues I might expect to hit if I were to try and plough ahead with making a GameActivity based app that used winit and probably egui to start with. In case I did go ahead any get something working it would also be interesting to hear if there would be any interest in accepting changes to support alternative Activity subclasses.

[dvc94ch]

The winit android backend is built around ndk-glue, which is NativeActivity specific. Is NativeActivity deprecated now?

[rib]

Maybe not for all use cases, but it does seem that NativeActivity is rather minimal and although it can serve ok for simple demos I'm not sure NativeActivity is necessarily the best general purpose foundation.

On the AGDK page they say:

Note: Previously NativeActivity was the recommended class for games. GameActivity replaces it as the recommended class for games, and is backwards compatible to API level 16.

GameActivity is at least a more recent up-to-date native abstraction aimed at game development use cases which might not cover everyone's needs but compared to NativeActivity then it looks like they provide more flexible / comprehensive native APIs (and looks like they are also backwards compatible with the ndk-glue) so all other things being equal it does look like it could be a better general foundation.

I haven't tried using GameActivity yet but I found that the AGDK libraries (including gameactivity) are also available via JetPack here: https://developer.android.com/jetpack/androidx/releases/games

I also found this samples repo in case it's of interest to look at: https://github.com/android/games-samples/tree/main/agdk

Thinking about winit, then I also wonder whether it would make sense to use their frame pacing and controller input libraries too.

[dvc94ch]

so GameActivity isn't part of the android.jar right? this means cargo-apk would need a maven package manager too.

[dvc94ch]

I'd prefer maintaining a RustActivity based on NativeActivity or GameActivity ourselves [0] and support "NativeService"s while we're at it [1]. However this is quite a bit of work. If you want to go the GameActivity/maven route there's a maven package manager in [2] which I built for building flutter apps without gradle.

• [0] vendor NativeActivity #151
• [1] https://github.com/rust-windowing/android-ndk-rs/issues/128
• [2] https://github.com/cloudpeers/xbuild/tree/master/mvn

[rib]

I think it would either require fetching from Maven or else it's possible to download the releases as zip files (with .aar libraries and pre-built native libs + C headers) here: https://developer.android.com/games/agdk/download

See here for more build integration details though: https://developer.android.com/games/agdk/integrate-game-activity they have tried to help make it possible to integrate with native build toolchains like cmake. One thing I'm not really familiar with is their 'Prefab' mechanism for bundling the android_native_app_glue source/headers as part of the GameActivity .aar.

Here is the implementation for GameActivity itself: https://android.googlesource.com/platform/frameworks/opt/gamesdk/+/refs/heads/master/GameActivity/ (including the android_native_app_glue code)

Although I had seen issue #151 (and I was initially going to raise this question as a comment there) I ended up making a separate issue instead.

From looking at the AGDK it seems like they have multiple libraries that could be of interest / useful for native Rust applications / rust-windowing beyond just the GameActivity - such as the frame pacing, the input method support and controller support etc as well as offering improved compatibility and options for integrating with jetpack libraries by being based on AppCompatActivity.

Considering the breadth of functionality that seems to be actively supported by the agdk I'd be inclined to want to leverage and build on that work instead of re-implementing something similar if it can be helped. The GameActivity class itself and the native glue code could be reimplemented but I guess it would also be possible to build on their glue code directly in a way that can easily track upstream updates. If it turned out that additional features were required then I guess they could be handled as a subclass of GameActivity and by extending the set of bindings on the native side.

Out of curiosity I did take a quick look at putting their glue code into a rust crate to build via the cc crate and actually hit a few compilation errors - which isn't exactly a great sign :/ I ended up filing this issue:
https://issuetracker.google.com/issues/229997306

Yeah dealing with Maven repos and some amount of Java seems inescapable for any non-trivial Android development. I noticed you had recently implemented some maven support for xbuild which looked interesting.

Looking at winit it depend on ndk-sys, ndk and ndk-glue and I suppose it's just ndk-glue that would need updating. The entrypoint is renamed to GameActivity_onCreate instead of ANativeActivity_onCreate which would also affect the macro used for marking the entrypoint.

Initially I was thinking of seeing how far I get with making a crate that's compatible with the ndk-glue API except based on GameActivity and maybe see how far I get with convincing winit to use that.

[rib]

As a follow up to this I ended up building a glue layer for GameActivity that can be found here:

https://github.com/rib/agdk-rust/tree/main/game-activity (the README there also has a bit more context/info)

I've also built a corresponding branch of winit that uses this glue in the Android backend:
https://github.com/rib/winit/tree/agdk-game-activity

For initial testing I've made:

• A minimal application that just verifies running a mainloop only based on the glue API itself:
  https://github.com/rib/agdk-rust/tree/main/examples/agdk-mainloop
• A minimal winit-based application that demonstrates drawing a triangle with wgpu; demonstrating handling lifecycle events and updating surface state each time we get a new native window: https://github.com/rib/agdk-rust/tree/main/examples/agdk-winit-wgpu.
• A minimal egui based application here: https://github.com/rib/agdk-rust/tree/main/examples/agdk-egui (which notably required some upstream changes to account for how native windows are only valid between resumed/paused events)

Thinking about enabling winit to support alternative Android glue layers it would be good to be able to discuss some options for normalizing our APIs, so then it might be possible to choose a glue crate as a winit build feature perhaps.

Initially I had planned to simply use the ndk-glue API so that my crate would be able to drop in as an alternative implementation for winit but as I progressed I realised that wasn't going to work out.

The first notable difference I ended up with was that I made poll_events() take a FnMut closure as a way to maintain the ability to surround any event handling callback with pre- and post- work internally. This is in part because that matches how the underlying android_native_app_glue works but also it does seem like a good technical solution for allowing transparent synchronization between the native thread and the Java main thread where required (such as for window termination or save-state events). As it is currently ndk-glue documents that for some of the events the consumer should take a certain lock to try and ensure synchronization where I think there may currently be a race condition. This approach is also reliant on the user of the API correctly handling those synchronization details manually. I should probably open a separate issue to discuss whether there is a race here currently, but do wonder if it could be good to consider an API that takes a closure instead so that any synchronization that's required can be encapsulated in the glue layer instead.

Something else notable that I did differently was to define an entry point ABI that is an extern "C" android_main()function and to also expose a globalAndroidAppstruct and API (accessible viagame_activity::android_app()). This doesn't preclude creating convenience #[main]macros that might do things like configure logging but at a low level, the only requirement is to export thatandroid_mainfunction as an entry point. In particular this symbol isn't inherently tied toGameActivityorNativeActivitywhich differs to howndk-gluecurrently has a#[main]that defines aNativeActivity_onCreatefunction that in turn needs to call aninit()function which is only really applicable toNativeActivityand also less ergonomic perhaps thanandroid_main` in case you want to forgo using the macro.

I think there are quite a few other aspects that would be interesting to discuss but it would be great if you might get a chance to look at what I have so far and then I can hopefully follow up with more details / thoughts if interested.

Although you commented that you'd like to build a custom RustActivity and that's not what I've been looking at here I think this work could potentially feed into any future effort to create a RustActivity later.

[zduny]

Would it be possible for ndk-glue to not have strict dependency on any type of Activity?

Just have it generate appropriate entry points, mechanism to pass surface to Rust for graphical applications and then let the user create Android project himself, simply document how it's supposed to be integrated with Rust (how to link native library, what to call and where).

IMO current "fixed" approach where everything is generated for you "behind the scenes" without your control is just way too inflexible for practical applications.

[MarijnS95]

Just have it generate appropriate entry points, mechanism to pass surface to Rust for graphical applications and then let the user create Android project himself, simply document how it's supposed to be integrated with Rust (how to link native library, what to call and where).

That sounds a lot like what you can achieve with Mozilla's Rust-Android-Gradle plugin, which is designed with this use-case in mind - cargo-apk and ndk-glue have totally different goals 😉

[zduny]

Mozilla's Rust-Android-Gradle seems to be only concerned about building Rust code.
It does not seem to do anything about the "interoperability (...) with the Android framework" part.

Focusing only on Rust-only applications is just too limited of a goal considering how Android ecosystem operates IMO.
Especially since ndk-glue seems to be influencing how winit and probably wgpu are shaped.

I admit I haven't looked too much into ndk-glue (because of the problems mentioned in this issue), but based on this project android-activity I can assume reimplementing its features for the use with other Activities isn't really that straightforward.

And that's a shame cause it makes a task like "adding 3D graphics to an existing Android app with the use of Rust and wgpu" (probably a common potential future use case of Rust on Android) unnecessarily difficult and confusing when it doesn't have to be.

I'm just saying developers would greatly appreciate if ndk-glue was adapted to more general use case.

[MarijnS95]

Mozilla's Rust-Android-Gradle seems to be only concerned about building Rust code. It does not seem to do anything about the "interoperability (...) with the Android framework" part.

Because there's no trivial way to define that. ndk-glue, and android-activity both allow you to have a pure Rust app running on Android, with a Rusty mapping to NativeActivity/GameActivity. winit piggy-backs on that by implementing the Activity lifecycle in an event loop.
Anything else and you're mostly on your own determining what APIs - over JNI calls - you're going to need to interface between a Rust and Java part (and rust-android-gradle really only helps building a Rust library and embedding it into the APK).

The ndk crate exists to give you access to Android framework parts in Rust, which are typically passed over the JNI boundary as jobject. I recently cobbled together a project that uses a Java Activity and merely calls into Rust for some rendering work.

Focusing only on Rust-only applications is just too limited of a goal considering how Android ecosystem operates IMO. Especially since ndk-glue seems to be influencing how winit and probably wgpu are shaped.

Feel free to design and contribute and implementation.

I admit I haven't looked too much into ndk-glue (because of the problems mentioned in this issue), but based on this project android-activity I can assume reimplementing its features for the use with other Activities isn't really that straightforward.

You're recommended to look into ndk-glue - it is really bare-bones and mostly offloads mission-critical code to winit. The crate is long overdue for an overhaul, but @rib beat me to it by yanking it out and replacing it with C++ instead, so it has some catching-up to do.

In any case, I recently started working on the idea of having a custom Java Activity specifically for ndk-glue, that implements some of the currently-missing (from both NativeActivity and GameActivity 🤭) callbacks users have been requesting. The interface between the Java and Rust part sounds a lot like what you're asking for (and could be a verbatim copy of NativeActivity/GameActivity on both Java and native part). I want to ship a precompiled .class file with the Rust crate so that it works out-of-the-box in a pure-Rust environment like ndk-glue + cargo-apk currently does, but also provide users some way to extend the Java class and add extra entrypoints between Java and Rust if they so desire.
Alas, progress recently stagnated while trying to work out the current winit + android-activity situation in a way that isn't destructive to ndk-glue.

And that's a shame cause it makes a task like "adding 3D graphics to an existing Android app with the use of Rust and wgpu" (probably a common potential future use case of Rust on Android) unnecessarily difficult and confusing when it doesn't have to be.

There are a multitude of ways to do this. A separate pure-Rust Activity (with a Java activity acting as the launcher)? Or a Java Activity calling into Rust for various parts of 3D rendering and game management?

I'm just saying developers would greatly appreciate if ndk-glue was adopted for the more general use case.

Fwiw you're the first one requesting this.

[zduny]

There are a multitude of ways to do this. A separate pure-Rust Activity (with a Java activity acting as the launcher)? Or a Java Activity calling into Rust for various parts of 3D rendering and game management?

The issue is that it's not uncommon for Android applications to have a SurfaceView inside a more complex layout - so SurfaceView doesn't take the whole space of Activity.

So, I guess in my ideal implementation ndk-glue/winit/wgpu (btw. that's the main confusing point to me at the moment - it's really hard to understand how interdependent they are and what crate does what exactly in the context of an Android application) shouldn't be concerned by the Activity itself but wrap around some custom SurfaceView - so this "custom SurfaceView" would be our "Window".

Then a developer could put (in Java/Kotlin) this "custom SurfaceView" wherever he wants to, then based on ndk-glue documentation would pass all the required events/input/lifetime events/whatever from Activity to this "custom SurfaceView".

I don't know if implementation like that is possible, but it would be nice.

And of course, by default ndk-glue/cargo-apk could still generate some default custom Activity wrapping around this custom SurfaceView doing all the plumbing in its implementation (or just have an alternative implementation for the NativeActivity/GameActivity/etc.).

EDIT: For the sake of completeness I'd also like to note that it's possible to have multiple SurfaceViews inside one layout (inside a single Activity).
So, I believe in that proposed above hypothetical implementation multiple SurfaceViews would map to multiple Windows in winit.

[torokati44]

... it's not uncommon for Android applications to have a SurfaceView inside a more complex layout - so SurfaceView doesn't take the whole space of Activity.

That's definitely what I want to do (and am already doing with @rib's android-activity crate, and the relevant winit PR/fork) for the Ruffle Android app.

Then a developer could put (in Java/Kotlin) this "custom SurfaceView" wherever he wants to, then based on ndk-glue documentation would pass all the required events/input/lifetime events/whatever from Activity to this "custom SurfaceView".

Yep, this would be super nice. I'm currently doing this by overriding (mostly copying) the method that constructs the SurfaceView.

Would also be great if the "inner window dimensions" (both position and size) would describe this SurfaceView on the screen, and if input (touch/mouse) events were mapped into it with correct coordinates.

EDIT: I don't care much how all of this is done under the hood as long as it's not too hard to use, and wgpu works with it.

[dvc94ch]

@rib @MarijnS95 what's the status of your efforts to replace ndk-glue? Seems like both your efforts use gradle instead of cargo-apk/xbuild? I proposed using dioxus/xbuild for building a mobile app at my current company, but hit some issues getting wry working without gradle [0]. I need to come up with a reasonable solution this week or I might end up having to do react native or some other crap.

• [0] Android support tauri-apps/wry#21

[rib]

hey @dvc94ch I'd say at this point android-activity is hopefully in good shape and I think rust-windowing/winit#2444 is ready to land.

I tried to make sure android-activity at least supports everything that ndk-glue supports but the API design was updated so that it could also work on top of GameActivity or potentially other activity implementations in the future.

I've generally tried to validate the crate with a decent number of examples here:
https://github.com/rib/android-activity/tree/main/examples

It should also solve a few issues mention in the comment here rust-windowing/winit#2444 (comment)

In terms of build system the crate can still be built with cargo apk, like ndk-glue if you're only using NativeActivity but if using GameActivity that at least involves linking with a pre-built class library and compiling an Activity that subclasses GameActivity.

Personally I prefer to just bite the bullet and use Gradle for anything involving compiling Java/Kotlin and packaging for Android since it's such a de-facto standard (that's just my preference though).

If you see the samples here then all the NativeActivity examples show how to optionally build with cargo apk for convenience but otherwise they all recommend using cargo ndk to compile the cdylib library (and copy the library across to the gradle project), followed by calling ./gradlew build.

It means running two separate commands but that's been acceptable for me so far because it means Gradle doesn't need to be taught anything about Rust and vice versa cargo doesn't need to be taught anything about Android (except for cargo ndk) which I find simpler overall.

Not sure if that info helps or not.

I tried looking at the above wry issue to get a bit more context and I can see it's going to be a challenge looking to use a WebView from a standalone native Rust application but I guess it may be possible.

For the issue of the Looper it's interesting that WebView expects to be able to find a looper for the current thread - I had been half hoping I could experiment with using epoll directly instead of supporting Looper but as it is currently then android-activity does already attach a looper for the Rust main thread, since that's the basis for the event polling.

[dvc94ch]

Yeah, tried your examples today and see how it works. I'll try getting wry working with your android-activity and gradle and then see how I want to support it in xbuild. Xbuild can build and run all your native activity examples out of the box.

I understand @MarijnS95 and you had a disagreement about the C/C++ code. It's fine with me, the gradle issue bothers me more. I guess we need to be pragmatic, so I'm in favor of your winit PR.

[rib]

Yeah, it' unfortunate that we've had some tension but I hope that it can be water under the bridge. You may not have noticed but fwiw there's no longer any C/C++ code if using NativeActivity in android-activity, since: https://github.com/rib/android-activity/pull/35

[dvc94ch]

ah cool! I had indeed not noticed.

[MarijnS95]

@rib can we start closing this issue off, now that you landed your backend in winit and finally solved all the issues with ndk-glue? I mostly need help landing the right deprecation warnings and code removal on this repo, given a severe lack of maintainer presence here.

[rib]

Yeah I think it makes sense to close this issue now

Yeah maybe we can follow up re: helping with deprecation warnings etc here: #365