Input event timestamps

[ishitatsuyuki]

On most OSes, a timestamp is provided alongside with the input event. It could be exposed as an Option or we can just fallback to the time the poll have been executed.

[dam4rus]

Which event do you mean on Windows? You can get the timestamp of the last event returned by GetMessage with GetMessageTime. It's a bit tricky though, because it's not in system time. It also provides buffered mouse inputs through GetMouseMovePointsEx, which reports the time of each mouse event.

[ishitatsuyuki]

@dam4rus Oh well - didn't know that Windows have actually provided that, so that was an oversight on my side. Thanks for the insight.

@zegentzy your labels doesn't make sense from the beginning, this issue was originally about something on non-Windows; and it turns out that this is also possible on Windows. Please remove the "platform: Windows" label altogether.

[goddessfreya]

@zegentzy your labels doesn't make sense from the beginning, this issue was originally about something on non-Windows; and it turns out that this is also possible on Windows. Please remove the "platform: Windows" label altogether.

Sorry, my bad.

When I saw this:

On most OSes other than Windows, a timestamp is provided alongside with the input event. It could be exposed as an Option or we can just fallback to the time the poll have been executed.

I interpreted it as currently all the platforms but windows are exposing a timestamp somehow, and windows should start exposing one too, or fallback to a None if it is not available.

No clue why I read it as that. My bad.

[Osspial]

Skimming through SDL's source code, it looks like they compute their timestamp with the SDL equivalent of calling Instant::now() when an event gets delivered, which doesn't seem very different from what our users would do to get the timestamp anyway.

I'm not necessarily going to say we shouldn't add this, but what utility does exposing event timestamps have for downstream users? The fact that SDL computes the event timestamp itself rather than asking the OS makes me wary about whether or not it's necessary, but SDL doesn't always make the right call and I may very well be missing something here.

[OvermindDL1]

If the user code causes the program to freeze for, say, 3 seconds, and someone pressed, say, 'a' twice with a second in between, they'd be able to get that information is why most OS's give timestamp information with the events. I'd at least emulate that even if it would be inaccurate on windows.

[Osspial]

@OvermindDL1 Ah, that's true! I'd definitely support adding these, then.

One caveat is that, on Windows at least, GetMessageTime is only accurate to the millisecond. If we expose this through the Instant API, we'd have to document that it can't be relied on for accurate measurements.

The only question I have now is where in Event should this be exposed?

[Ralith]

IMO this should be present on every Event. A simple approach is to add the timestamp as an extra argument to the run closure. Alternatively, we could introduce something like struct EventInfo { event: Event, timestamp: Instant }.

[Osspial]

Adding a timestamp to the run closure seems like the cleanest solution to me.

[superdump]

So if I understand correctly, you’re suggesting that an additional argument be added to the run closure here: https://github.com/rust-windowing/winit/blob/master/src/event_loop.rs#L152 and that the call sites pass the time stamp. Right?

[Ralith]

That's correct.

[kettle11]

Summarizing some API info here so that it may help anyone who wants to implement this:

MacOS

NSEvents have a timestamp property that can be used: https://developer.apple.com/documentation/appkit/nsevent/1528239-timestamp?language=objc
The property is a f64 representing seconds since system startup.

Windows

GetMessageTime: https://docs.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-getmessagetime
The returned value is an i32 (c_long on Windows) representing milliseconds since startup time.

Web

Each event has a timeStamp property that can be used: https://developer.mozilla.org/en-US/docs/Web/API/Event/timeStamp

For web I would be suspicious that browsers may timestamp based on when they enqueue the event, but that would have to be tolerated.

I don't know about other platforms winit supports.

[ishitatsuyuki]

I decided to look into this, and here are some additional insights to add to #1194 (comment):

macOS

f64 itself is capable of representing very high precision values of the timestamp, but it's unknown whether the OS is coded to accurately capture these. I'd appreciate if someone tests it.

Windows

GetMessageTime depends on OS ticks, which is notorious for being inaccurate. (reference) IMO not worth using, use a timestamp synthesized with Instant::now() instead.

X11

X11 timestamps 1. have server reset time as the basis (and provide no method for accurate conversion) and 2. has millisecond granularity. IMO not worth using, use a timestamp synthesized with Instant::now() instead.

Wayland

Core protocol provides millisecond granularity timestamps, which are not worth using. An extension provides nanosecond granularity timestamps, which are useful, especially given that Linux have extremely high accuracy dealing with interrupts and timers.

The problem with the extension is that the clock domain is undefined per spec. Practically everyone are using libinput, which uses CLOCK_MONOTONIC internally. (source)

Overall a lot of OSes have poorly designed APIs that provides timestamps where their precision are nowhere near useful, and applications had better count on synthesized timestamps in an event loop that is not blocked by rendering operations (i.e. events not being compressed or delayed otherwise). But on macOS and Wayland it might be worth implementing this.

[Ralith]

Given that games routinely render with frame intervals much larger than a millisecond, I'm not sure I agree that 1ms granularity is categorically "nowhere near useful"; it is trivially much more precise than calling Instant::now() in that context, for starters.

[ishitatsuyuki]

Practically, jitters under the 1ms range is going to be unnoticeable, so I'm going to take back the claim of "nowhere near useful". There are still cases that might be problematic though, e.g. handling mouse acceleration in the application (amplifies jitter) or dealing with devices having a polling rate that does not divide 1000.

Given that games routinely render with frame intervals much larger than a millisecond,
... it is trivially much more precise than calling Instant::now() in that context, for starters.

This assumes that rendering happens on the same thread as input handling (which has to be the main thread in winit), which is not necessarily the case. As long as one don't place blocking work on the input thread, timestamping when the event is received is still going to have a better precision than the millisecond-granularity timestamps sent by the OS. For the reference, there are real-world implementation that does this, like MouseTester using QPC.

[Ralith]

I wouldn't describe rendering as "blocking work", but on e.g. macOS it must happen on the same thread that handles input, and regardless of platform games tend to be built this way in practice. If winit exposes the OS timestamp and documents any issues, then downstream code has the option to choose according to their requirements.

[maniwani]

Are there any major blockers to implementing this feature?

I'm trying to resolve bevyengine/bevy#6183, which is closely related this issue. Resolving the problem requires granular timestamps for input events (i.e. not aliased to the nearest frame). 1ms granularity is enough for that purpose.

I planned to move application updates into another thread so they don't block the winit event loop, that way timestamps synthesized with Instant::now() would have very high granularity, but winit passing along timestamps seems like a more elegant solution (my solution can't be applied on wasm32, for example).

Do the OS event timestamps described earlier come from the same monotonic clock as Instant on each platform? If so, I'd be willing to work on the implementation.

[krakow10]

what utility does exposing event timestamps have for downstream users?

Here's a good reason to add timestamps: if you are busy doing work in the event loop thread, how will you know at what time the events that built up came in? For example, without timestamps there's no way to trace the exact space-time path the mouse took while the event loop was busy (which is important for my game). Ideally the event loop thread is never busy, but that's not guaranteed and multi threading is hard!

[kchibisov]

Here's a good reason to add timestamps: if you are busy doing work in the event loop thread, how will you know at what time the events that built up came in? For example, without timestamps there's no way to trace the exact space-time path the mouse took while the event loop was busy (which is important for my game). Ideally the event loop thread is never busy, but that's not guaranteed and multi threading is hard!

Yeah, I understand this myself and I use timestamps for the exact same reason inside winit.

Though, not each event should have a timestamp, and I'm not sure how to make them unified... But the issue is open and I'd like events to forward timestamps when they have it from the OS as well. You can also sync audio with them to the key presses, etc.

[expikr]

Practically, jitters under the 1ms range is going to be unnoticeable, so I'm going to take back the claim of "nowhere near useful". There are still cases that might be problematic though, e.g. handling mouse acceleration in the application (amplifies jitter) or dealing with devices having a polling rate that does not divide 1000.

Given that games routinely render with frame intervals much larger than a millisecond,

... it is trivially much more precise than calling Instant::now() in that context, for starters.

This assumes that rendering happens on the same thread as input handling (which has to be the main thread in winit), which is not necessarily the case. As long as one don't place blocking work on the input thread, timestamping when the event is received is still going to have a better precision than the millisecond-granularity timestamps sent by the OS. For the reference, there are real-world implementation that does this, like MouseTester using QPC.

FYI, mousetester is singlethreaded with a blocking message loop, all it does is to do nothing other than fetching rawinputs, so probably not a good model to emulate for games wanting precise timestamping.

If you want precise timestamping, you must use a separate thread and create a dedicated invisible window target so that its message loop only receives raw inputs.

[krakow10]

Though, not each event should have a timestamp

Why not? Are they not all given one at the OS level? Is there some special event types that timestamps don't make sense for? I can't see how including it with every event could be anything but a good thing, for games at least.

[kchibisov]

Yes, not all events will carry a timestamp with them. On wayland you can grasp with this https://wayland.app/protocols/wayland .

We could still probably compute relative in some cases.

[krakow10]

Yes, not all events will carry a timestamp with them. On wayland you can grasp with this https://wayland.app/protocols/wayland .

We could still probably compute relative in some cases.

I don't grasp how I can use this page to see which events do and don't correspond to events with timestamps. However, I do see something that I like: https://wayland.app/protocols/input-timestamps-unstable-v1 How can I see which compositors support this? If I was so inclined, how would I go about adding this to winit?

[kchibisov]

I don't grasp how I can use this page to see which events do and don't correspond

Just look for timestamp, it's present on some events.

@krakow10 it says it's supported only on weston.

[maniwani]

I decided to take a look at this. Commenting here to share some findings.

For my own use, I don't want to deal with timestamps that aren't Instant, so I tried to synthesize Instant timestamps for all window and device events using their OS-provided timestamps.

If you take two pairs of OS and Instant::now() timestamps (both clocks sampled at about the same time), you'll have two ranges describing the same span of time. If you then take the OS timestamp of an event that's in between the two OS samples, you can calculate the corresponding Instant by linear interpolation (which should closely approximate the true value¹).

I tried this on Windows (which has the. most. obtuse. timekeeping API) and discovered that the resolution of event timestamps (GetMessageTime) is 16ms in the best case. From my testing, event timestamps are sourced from a counter (GetTickCount) that normally updates at 64Hz. I thought I could increase that rate since Windows has a method to set the resolution of its periodic timers, timeBeginPeriod, but instead I observed that GetTickCount is not affected by it.² 🙃

So on Windows at least, there doesn't seem to be a way to get a higher resolution short of moving your application logic into another thread so it doesn't block the event loop. But if you do that, you can just call Instant::now() in the event handler.

Do the OS event timestamps described earlier come from the same monotonic clock as Instant on each platform?

Just to answer my own question, this is a definite "No." for Windows. I think it's a "Yes." for web and for Linux (I believe evdev lets you choose the clock source it uses. I don't know if libinput is the same way, but if it uses CLOCK_MONOTONIC, that would at least match Instant). I don't know about macOS/iOS.

Footnotes

1. But limited by the lower resolution of the two clocks. ↩

2. I also found some other supporting docs saying the same thing. ↩

[mickvangelderen]

For my application I want to use a fixed timestep. Multiple simulation steps can be performed between rendered frames. This occurs when rendering takes more time than is available (v-sync rate). In order to respond to inputs properly I need to associate the input with the timestep in which it occurred. I don't want an expensive frame to impact how long a key is considered to have been pressed, for example. Judging from this discussion this seems like a common use-case.

Setting up a thread to associate timestamps with events, and then forwarding those events to the rendering thread is a decent solution, but it does requires a bit of know-how and some code to achieve. On some platforms there may be a way to get timestamp information from some OS events and perhaps you need to disable batched event delivery. Having every developer figure that out by themselves for every platform seems counter to what winit attempts to provide.