Stream is Lagging after sleeping device for a while, then waking device

[thiccaxe]

ios 17 dev beta 7 - might be fixed in final release

IIRC previous behavior was that the phone would send a black screen on shutoff but now I think it stops sending video updates when you press power button. after 5-6+ seconds wake the phone and the stream is delayed and laggy.

This is not the case on airplay2, on an apple tv hd tvOS 16, where everything works as expected.

[thiccaxe]

In these logs I connected with screen mirroring as per normal, then periodically turned off and turned back on phone several times.

2023_09_02_turning_phone_on_and_off_screen_mirror_lag.txt

Don't know the exact cause. But it seems like the display end stream expects data but none is provided. Based on some tests with a stopwatch, the stream resumes after the data has been back-filled. Essentially, if the screen is off for 5 seconds, it will take 5 seconds for the stream to catch up again in the gstreamer window.

I will test audio and post another update...

[thiccaxe]

I tested audio, and similar things seem to be happening. note that this is all on 21A5319a

[thiccaxe]

OK, I have found a solution, I am pausing the gstreamer stream when the client sends the updated window data, then start again on the next video data packet. As well as resetting the base time.

But this does not solve all the problems, like from the VLC for ios app - it also yields a black screen. I will do more research, and submit a pull request soon

[thiccaxe]

This issue seems to be more complicated. It seems like airplay will (sometimes) not send updates if the content on the screen does not change. In this case, the solution doesn't work. Additionally, it fails in case that the device goes to sleep on its own (user doesn't press sleep/wake button manualy)

It seems a better solution is required. I will do more research.

[fduncanh]

For video, there are two gstreamer modes "sync = true" and "sync = false"

"sync = true" is the default: uxplay or "uxplay -vsync" this matches audio and video timestamps from the client.
EDIT: this drops video frames which are "too late"

"sync = false" is the older method: get it back with "uxplay -vsync no" this live streams and ignores timestamps, It times audio using 44.1 kHz and video by fps. EDIT: here video frames dont get dropped if they are late.

[fduncanh]

Is there something I should look for in the 1.17 public beta?. I also have another ipad on 1.16

What should I do to see your issue, e.g. while a you tube movie is playing?

[thiccaxe]

It affects iOS 16 too:

• start screen mirroring
• press sleep/wake button
• wait 5-10 s
• wake device
• observe video stream does not update for equivalent time

[fduncanh]

OK so this is not a 1.17 issue.

Have you tried using uxplay -vsync no ?

(The vsync option wont affect the raop debug output, it only affects gstreamer what gstreamer does)
.
can you update your debug trace to add annotations showing what to look at? (or post excerpts in a comment)

[fduncanh]

For vsync mode, it would be easy to add a delay option (+ or -) to both audio and video timestamps.

[fduncanh]

OK I think I see it.

Yes its only there with the vsync option which I made the default as of UxPlay 1.64.

Maybe I should go back to -vsync no as the default ? (i.e. behavior prior to 1.64)
Vsync drops video frames that are "too late" (time stamps would be later that "now" when rendered)

[thiccaxe]

Was there any reasoning behind the default video sync setting?

Anyway, I think it's better to investigate the issue, because vsync could be useful.

[fduncanh]

Yes, it guarantees that video and audio stay coupled. It is important for low power systems without hardware h264 decoding.

Each frame in the video stream has an NTP time stamp (seconds since jan 1, 1900 to jan1, 1970 plus time since client last boot) The NTP synchronization between client and server (abandoned in AirPlay 2) converts that to server Unix time on the client.

The audio sends "rtptime" based on the 44.1kHz audio encoding (This is iTunes mechanism). shortly after the AAC-ELD audi stream starts, the client sends signals at intervals with a client NTP time and client rtptime of the same instant.
This allows audio rtptime to be kept from drifting relative to the equivalent video time.

So within a few seconds of the audiostream starting, audio and video have timestamps (in nanoseconds) based on client ntp time in seconds relative to an arbitrary epoch. This allows lipsync for ever. After a mirror connection starts, the offset betweeen client ntptime and unix time is computed, and used to adjust the timestamp offsets to match server time.

What I hadnt realized for a long time was that the way antimof implemented gstreamer support used "sync = no" which makes no use of timestamps at all. This is "uxplay -vsync no" mode, and is just live streaming: play audi when it arrives, play video when it arrives. This works if the video decoding is fast enough. If not video gets more and more delayed.

In vsync mode video frames with timestamps (indicating when they should be played) later than "now" get dropped.
(audio has about a 5 second latency so is always decoded in time)

One sees the video frames getting dropped when GST_DEBUG=2 is active.

[fduncanh]

• The raop subsystem delivers audio and video frames to uxplay.cpp by callbacks "audio_process" and "video_process"

• each audio and video frame contains the unencrypted data, plus the "ntp_remote" timestamp, and its "ntp_local" value according to the ntp synchronization of client (remote) and server (local) clocks. (The audio data also includes the rtptime and secnum data from which the audio frame's ntp_remote timestamp was obtained)

• each of these callbacks then send the data plus (modified) time stamps on to the gstreamer audio and video renderers

• i'm not totally sure I've done the right things when "sync=true". I compute the offset between local and remote time
  once when the first (audio or video) packet of the connection arrives, so ignore ntp_local of the packet (I just add the offset to ntp_remote). I also use the timestamps as PTS (presentation timestamps) rather than DTS (decoding timestamps).

*Any modification can be done completely in uxplay.cpp and renderers, without touching raop code in lib

• "sync = false" gstreamer mode ("uxplay -vsync no") ignores the timestamps completely. I believe it renders as soon as decoding is done, and if data arrives when the buffer that holds it before decoding is full, just discards it (??)

[fduncanh]

https://gstreamer-devel.narkive.com/v8ic4w0S/usage-of-sync-false-in-rendering

[thiccaxe]

I compute the offset between local and remote time
once when the first (audio or video) packet of the connection arrives

In my fix I had to repeat this as well. I think we need to reset the time when raop receives a window dimensions update packet (which is why RAOP code needed to be modified) (unencrypted coded data). Let me upload my changes to git

[thiccaxe]

thiccaxe@660a2dc

this isn't a great solution though, its more of a soggy bandaid.

[fduncanh]

can you identify the window dimension update packet in the debug trace?

I did some testing after issues raised in #169.
I recall I found that using the ntp remote timestamps (the audio one computed from client data only) as PTS was much more accurate than using the conversion to "ntp local" using ntp, which introduces jitters.

When the client is send video at 30fps, the client timestamp increase by 0.03333xx secs per frame.
The rtp time increase according to 44.1kHz, and the client peridiocally sends an rtptime+ntptime pair that allows audio rtptime to be converted to video ntptime, using an offset averaged over the previous 8 sync signals.

• The general opinion in Lip sync is that the audio is most important, dropping some video frames is not so important.
• I need to see if some latency adjustment might be useful (delay the PTS of both video and audio a bit)

[fduncanh]

Just saw you PR. looks like a solution.

Did you look at:

https://gstreamer.freedesktop.org/documentation/application-development/advanced/clocks.html?gi-language=c

[thiccaxe]

Just saw you PR. looks like a solution.

it has some bugs that can pop up unfortunately (eg. when the device goes to sleep by itself). I'll took a look at the link, I think this section:

Synchronization is now a matter of making sure that a buffer with a certain running-time is played when the clock reaches the same running-time. Usually, this task is performed by sink elements. These elements also have to take into account the configured pipeline's latency and add it to the buffer running-time before synchronizing to the pipeline clock.

and The clock object needs to report an absolute-time that is monotonically increasing when the element is in the PLAYING state. It is allowed to pause the clock while the element is PAUSED. is particularily relavent

[fduncanh]

using GST_DEBUG=2
it seems the frames are getting dropped when the client restarts because they are "too old" I thing the gstreamer
clock needs to be paused during sleep.
The issue is how to detect sleep.

Maybe keep track of time since last data arrived?

[thiccaxe]

I think we need to some how tell gstreamer that these frames were skipped?

[thiccaxe]

Yeah, I think we need to pause gstreamer properly.

Maybe keep track of time since last data arrived?

seems like a good idea

[fduncanh]

@thiccaxe can you submit your fix as a PR or should I just copy from it?

I think its good after testing.

[thiccaxe]

Yeah, will submit a pr

[fduncanh]

PR is merged!

Thanks!