Reduce latency by 1 frame

[rom1v]

This PR contains unrelated commit after I pushed force it (I cannot change the base branch). The relevant commit is 63af7fb.

To packetize the H.264 raw stream, av_parser_parse2() (called by av_read_frame()) knows that it has received a full frame only after it has received some data for the next frame. As a consequence, the client always waited until the next frame before sending the current frame to the decoder!

On the device side, we know packets boundaries. To reduce latency, make the device always transmit the "frame meta" to packetize the stream manually (it was already implemented to send PTS, but only enabled on recording).

On the client side, replace av_read_frame() by manual packetizing and parsing.

https://stackoverflow.com/questions/50682518/replacing-av-read-frame-to-reduce-delay
https://trac.ffmpeg.org/ticket/3354

---

I'm opening a pull request to get more tests before merging (the changes are quite sensitive), to avoid regressions.

To confirm that it actually reduces the end-to-end latency by 1 frame, I created a video which increments a number at 30fps (every 33.3ms):

# requires ffmpeg and imagemagick
for i in {0..999}; do convert -size 400x400 xc:White -gravity Center -pointsize 200 -font DejaVu-Sans-Mono -annotate 0 $i $(printf '%03d' $i).png; done
ffmpeg -framerate 30 -pattern_type glob -i '*.png' -c:v libx264 -vf format=yuv420p 30fps.mp4

I run this video in VLC on Android, and start scrcpy.

I compared the current master with this PR for both 1080x1920 (just running scrcpy) and 448x800 (running scrcpy -m800).

1080x1920

on master

End-to-end latency is about 3 frames (~100ms).

on lowlatency

End-to-end latency is about 2 frames (~67ms).

Once a packet is received by the client, on average, the frame is rendered 12ms later.
(with "skip frames" disabled, i.e. --render-expired-frames)

448x800

on master

End-to-end latency is about 2 frames (~67ms).

on lowlatency

End-to-end latency is about 1 frame (~33ms).

Once a packet is received by the client, on average, the frame is rendered 8ms later.
(with "skip frames" disabled, i.e. -m800 --render-expired-frames)

---

For now, I preserved the compatibility with prebuilt server v1.9, so you can just:

git fetch origin && git checkout lowlatency

then build as usual.

For windows users, I built a binary (replace the one from v1.9): scrcpy.exe.
SHA-256: 6b8eae4eb4df811b6aa194320f00b51102cf3a254e432f353963863c82f362ea

Thank you for your feedbacks 😉

[rom1v]

I reworked it and merged in dev branch: 63af7fb.

Based on that, I also made the recorder asynchronous, so that it never delay the decoder on blocking I/O: 35d9185.

Tests welcome to find bugs before the next release 😉

[ghost]

Tried with scrcpy --bit-rate 1M --max-size 240 --max-fps 30 got 2 frame latency at 136x240, It's my device dependent, if it was faster than rom1v's unknown device, it would be 0 to 1 frame latency, without scrcpy, I would struggle with compressed image, inconsistent frame rate and image freeze spikes, thanks that scrcpy existed!!!!!! 😇

[rom1v]

Tried with scrcpy --bit-rate 1M --max-size 240 --max-fps 30 got 2 frame latency at 136x240, It's my device dependent,

Wow, that's a lot of latency for such a small definition.

if it was faster than rom1v's unknown device

The device I used in this test was a Nexus 5. The results are better with newer devices (like a OnePlus 7 Pro).

[ghost]

Actually x265 outperforms in everything x264:
0 ms latency x265, 12ms latency x264. x264 just sucks. Native 2400x1080 (the phone's screen is left one, the images are dark because it is 1/12000s so numbers don't look blurry)
x264:

x265:

[rom1v]

I'm not sure to understand the screenshots, but it depends a lot on the device and hardware encoder. For example, on a One Plus 7 Pro, the h264 encoder has a lower latency than the h265 encoder.

[ghost]

I'm not sure to understand the screenshots, but it depends a lot on the device and hardware encoder. For example, on a One Plus 7 Pro, the h264 encoder has a lower latency than the h265 encoder.

Basically i wrote a c++ program to display "i+1" number every 1ms, so i can take a photo and see how different are numbers on my phone and on my pc, basically latency.
With screen overclock to 77 hz (i wish there was a mod to trick android into rendering 120fps so the picture with scrcpy would be smoother), the frame time should be around 12ms, that's the latency of h264, however, h265 doesn't have any latency at all because numbers are the same (i mean of course it does, but the latency is smaller than 12ms)

It would be so much better, if h265 was the default one.

UPD: i just made a super slow motion video with both screens, it appears, that the latency is around 8-9ms, while using h265