Rework (split) Event::{Suspended, Resumed} to more clearly convey their meaning (on Android)

[MarijnS95]

The Suspended and Resumed events currently don't quite carry the load of "you must create and destroy your graphics resources created from RawWindowHandle ¹ within these events".

At the same time they are disjoint from Android's onPause and onResume lifecycle events (see also #2118), as the framework is "free" to not destroy the backing surface for its window even when the user switches away.
As far as I understand these events are used this way in iOS.

Then there exist WindowEvent::Focused(bool) events which correspond to https://developer.android.com/reference/android/app/Activity#onWindowFocusChanged(boolean), that itself isn't part of the holistic activity lifecycle either.

Finally these events don't correspond to a window being created nor WindowEvent::Destroyed, as the actvity itself generally stays alive. These events purely correspond to Android's need for managing the underlying Surface (a producer in a queue of images, not even directly the backing buffer of the window on screen), and IIRC map directly to the callbacks in https://developer.android.com/reference/android/view/SurfaceHolder.Callback.

There doesn't seem to be any similar system in the other backends, and again Resumed and Suspended don't quite relay this concept of surface handling. At the same time we are not relaying

Related

• Compiling on windows for android errors #1886 #2118: Tried to map the actual onPause/onResume lifecycle events to Suspended and Resumed, showing that there's clearly a naming conflict on these events;
• Documentation on Event::{Suspended, Resumed} is lacking #2185: Documentation lacking for these events, showing that they aren't descriptive;
• Consistently deliver a Resumed event on all platforms #2331: Raises the Resumed event across all platforms, and adds complicated "portability" documentation showcasing that the status-quo isn't the right way forward;
• Evolution of the Android backend #2293: More general overview of "all things wrong" concerning winits misunderstanding and lacking mapping of Android's lifecycle and various events/callbacks to winit callbacks.

CC @rib

More events to consider

(https://github.com/rust-windowing/winit/pull/2118/files#r777200781)

• onStart: No mapping
• onResume: Resumed
• WindowCreated ²: SurfaceCreated(RawWindowHandle?)
• WindowHasFocus: Focused(true)
• WindowLostFocus: Focused(false)
• onPause: Suspended
• onStop: No mapping
• WindowDestroyed ²: SurfaceDestroyed
• User presses back button Support Back button/KeyCode on Android #2304: CloseRequested (if there's no other back stack??)
• Destroy: WindowEvent::Destroyed?

And how do we deal with multiple Activity instances (multiple windows, really)?

Footnotes

1. Surfaces, native windows, oh my! ↩

2. ndk_glue naming, because NativeWindow == Surface in the NDK. ↩ ↩²

[rib]

My initial thoughts here are:

1. The existing Suspended/Resumed events may never be a perfect match for platform-specific lifecycle events but that might OK because they can still provide a portable mechanism for delineating where applications will be blocked from rendering - which is practically useful on both Android and iOS - regardless of the perfect alignment with lifecycle events.
2. The Resumed event (with Consistently deliver a Resumed event on all platforms #2331) can give us a practical place to recommend that applications initialize graphics state that helps in writing portable applications.
3. Instead of trying to split up the semantics of Suspended/Resumed events to try and get closer alignment with platform-specific lifecycle events I'd consider the tradeoff that this will add complexity in defining the meaning of these events and alternatively investigate how to add orthogonal events that may help with more platform-specific needs while keeping these more general-purpose events.

Overall I see that Winit is a portability abstraction and for something like this I wouldn't necessarily expect a perfect match with the underlying system events, which may be ok if they still solve a practical problem.

If we broadly define the Suspended/Resumed events as a mechanism for delineating when apps are paused/suspended from rendering I think these portable events can pretty much be kept as is; I would probably consider a few minor tweaks though:

1. On iOS I'd consider reporting Suspended when the application really transitions to "inactive" (currently the app is suspended when it's about to become inactive - and technically on iOS apps may still render in this phase)
2. On Android I'd make sure the backend Suspended was a strict logic OR between "lifecycle paused" and "surface view deleted" and Resumed should be a logical AND between "lifecycle resumed" and "surface view created"

Effectively trying to expose platform specific lifecycle semantics could end up being a pretty complex can of worms where I guess it may never be possible to come up with a super set of events that will cover the nuanced differences between platforms.

One idea I had the other day was potentially to expose an event like SystemLifecycleChanged which would intentionally be defined as a platform-specific event to avoid trying to define a platform-agnostic lifecycle model in Winit. This way there would still be the portable Suspended/Resumed events and then anything that really needs to track detailed lifecycle changes would have to have some amount of platform-specific code that understands the nuances of that platform.

Thinking of ways to more explicitly emphasize that applications should wait until they are Resumed before they initialize graphics state + windows (e.g. also avoiding changing the HasRawWindowHandle trait to return an Option<>) another idea I had recently was that we could potentially document that create_window() will fail before the application is Resumed and could also phase that in with warnings initially before looking to make it a hard error on all platforms. That might help lead to portable applications by default. It's still an extra step for Android portability to re-create surfaces for each Resumed event but tbh that's the easy part compared to getting applications to defer their graphics state initialization.

It could still also be reasonable to consider adding some extra events like WindowEvent::SurfaceEnable and WindowEvent::SurfaceInvalidated (tweaked naming from my initial SurfaceCreated SurfaceDestroyed idea while I'm trying to think how these could be defined in a way that seems less Android specific.) This doesn't necessarily need to affect the definition of the Suspended and Resumed events though.

[MarijnS95]

Instead of trying to split up the semantics of Suspended/Resumed events to try and get closer alignment with platform-specific lifecycle events

This is only in light of surfaces needing to be created/destroyed, which is an explicit action users of winit should perform. We shouldn't conflate resource management together with other lifecycle events if they're not strictly identical, that only needs to additional confusion.

1. On iOS I'd consider reporting Suspended when the application really transitions to "inactive" (currently the app is suspended when it's about to become inactive - and technically on iOS apps may still render in this phase)

Does the user need to destroy their surfaces on iOS here too (and to prevent being countered again: or leak it, as long a new surface is created in Resumed)?

2. On Android I'd make sure the backend Suspended was a strict logic OR between "lifecycle paused" and "surface view deleted"

When exactly would you fire this, then? Once for onPaused and once for NativeWindowDestroyed?

Other than that I agree that we don't need to strive for every platform-specific event under the sun, but I do feel that we can do a better job at making Suspended+Resumed and the surface management it implies more explicit. On the one hand you talk about this being too platform specific yet at the same time the goal is to implement these in a "platform agnostic" manner such that users writing desktop code know what to do and effortlessly have the unaltered winit loop working flawlessly on Android (and iOS).

Regarding create_window() failing prior to the Resumed event, that goes right against an earlier comment in #2331 that a window generally stays open (is not destroyed) during these events, just that their backing buffer disappears - which can lead to very different decisions in user code.

[rib]

1. On iOS I'd consider reporting Suspended when the application really transitions to "inactive" (currently the app is suspended when it's about to become inactive - and technically on iOS apps may still render in this phase)

Does the user need to destroy their surfaces on iOS here too (and to prevent being countered again: or leak it, as long a new surface is created in Resumed)?

As far as I know Android is the only system that effectively invalidates window surfaces automatically, so I think on iOS they can count on their surfaces remaining valid. They can maybe destroy surfaces while they aren't rendering (while inactive) to help minimize memory usage - but as far as I recall there's no requirement to do this.

For slightly improved consistency though, maybe it would make sense on iOS to map Suspended to being "inactive", so then Winit's Suspended and Resumed events can more simply be documented to delimit when applications are blocked from rendering.

[rib]

2. On Android I'd make sure the backend Suspended was a strict logic OR between "lifecycle paused" and "surface view deleted"

When exactly would you fire this, then? Once for onPaused and once for NativeWindowDestroyed?

It would be a really small tweak in the existing backed logic, that would essentially track the two things separately internally (i.e. set a flag for the lifecycle paused/resumed state and a flag for the window create/destroyed state) and then whenever either flag is updated the backend would potentially dispatch a Resumed event if state.lifecycle_paused == false && state.has_surface_view and emit a Suspended event if state.lifecycle_paused || state.has_surface_view == false. So there'd still just be one place for emitting either event.

[MarijnS95]

As far as I know Android is the only system that effectively invalidates window surfaces automatically, so I think on iOS they can count on their surfaces remaining valid. They can maybe destroy surfaces while they aren't rendering (while inactive) to help minimize memory usage - but as far as I recall there's no requirement to do this.

Is it common to do that to save on memory though? Otherwise I'm still inclined to have a very specific event solely related to surface creation and removal just to make intent completely clear. But it muddies the waters some when thinking about surface removal versus "you can only render within these events". Not sure if those should be considered the same or not...

For slightly improved consistency though, maybe it would make sense on iOS to map Suspended to being "inactive", so then Winit's Suspended and Resumed events can more simply be documented to delimit when applications are blocked from rendering.

You mean changing from the willresignactive bit to some callback that represents actually being inactive now? I didn't comment on that in the previous PR but that seems more consistent, yes.

[MarijnS95]

It would be a really small tweak in the existing backed logic, that would essentially track the two things separately internally (i.e. set a flag for the lifecycle paused/resumed state and a flag for the window create/destroyed state) and then whenever either flag is updated the backend would potentially dispatch a Resumed event if state.lifecycle_paused == false && state.has_surface_view and emit a Suspended event if state.lifecycle_paused || state.has_surface_view == false. So there'd still just be one place for emitting either event.

For the Suspended case there'd be only one place where it's dispatched, but this if would trigger for both the lifecycle_paused = true and native_window_lock = None case, so you'd send the event twice?

[rib]

It would be a really small tweak in the existing backed logic, that would essentially track the two things separately internally (i.e. set a flag for the lifecycle paused/resumed state and a flag for the window create/destroyed state) and then whenever either flag is updated the backend would potentially dispatch a Resumed event if state.lifecycle_paused == false && state.has_surface_view and emit a Suspended event if state.lifecycle_paused || state.has_surface_view == false. So there'd still just be one place for emitting either event.

For the Suspended case there'd be only one place where it's dispatched, but this if would trigger for both the lifecycle_paused = true and native_window_lock = None case, so you'd send the event twice?

to clarify here; the logic here would determine when we're "suspended" or "resumed" and the backend can also explicitly track that, e.g. via some state.is_suspended that lets it de-bounce and avoid re-emitting a redundant event (i.e. don't send a redundant Suspended event if state.is_suspended == true)

For reference though I did also include a note in the docs for #2331 that apps should be resilient to redundant events (even though they are easy to avoid here) just to reserve wiggle room for handling awkward, unforseen corner cases in backends and also just to be clear about what guarantees are provided by winit (re: #2185 where this specific question of redundant events was also discussed).

[MarijnS95]

@rib Regarding your explicit request to have logical operators between these events:

• Suspended be an OR between onPause and onNativeWindowDestroyed;
• Resumed be an AND between onResume and onNativeWindowCreated.

I don't think it's a good idea to generalize surface events together with the onPause/onResume callbacks and in turn leave developers wondering whether they should really create/destroy their implicit surface references. I'd have to check the semantics of this, but for the suggested OR case on Suspended, a user may run an app in split screen where focus is lost and rendering may stop, resulting in onPause():

Called as part of the activity lifecycle when the user no longer actively interacts with the activity, but it is still visible on screen.

So a graphics surface may very well continue to exist and be visible on screen, while destroying a graphics surface/swapchain on top of it could lead to unneeded corruption or black/empty surfaces.

In the Resumed case you may pay a performance hit for recreating surfaces and swapchains unnecessarily, on top of the exact same Surface. Perhaps even missing a frame in the case that onNativeWindowCreated is called well before onResume exactly to combat this (when an application did become invisible for a while and did give up its surface).
Then, for the AND case on Resumed you must track whether onNativeWindowCreated was last called, and recreate the surface in onResume if the user received a Suspended because of onPause and obediently destroyed their surface, without having a matching onNativeWindowDestroyed. But I guess you intended to do that anyway since you can't really be sure of the order in which onPause and onNativeWindowCreated are called.

Then finally, you mentioned above that Android is the only platform where surfaces are created and destroyed (iOS devs can rely on the surface to remain existing during the lifetime of the app), so there isn't much to generalize anyway. Creation and destruction only has meaning on Android, and could get separate - clearly defined - events for that.

[MarijnS95]

Looks like we're sticking with the current terminology, but at least everyone is on the same page now that this is where surface creation/destruction should happen.