Recorded or Displayed video puts some older frames after more recent frames – [feature request] option to select encoder #666
Comments
|
I was able to modify the client code and add logging statements, and I was able to determine that even though I still see the visual issue where periodically an image that would have been appropriate a fraction of a second earlier is briefly displayed, all of the PTS (Presentation Timestamp) values are being received in ascending order as they are read from the socket, and the AvPackets are processed in that same order. The bytes remaining read from the meta header for each frame matches the size of each AvPacket as it's being processed too. So I have to suspect that the problem is somewhere in the phone side; it seems to be producing the PTS values in ascending order, and without any excessive gaps in PTS value, so probably it's periodically putting out incorrect video data for some of the frames it's choosing to put out. It might be worth noting that I've tried recording this phone screen with a couple phone apps, such as APowerRec and AZ Screen Record, and they seem to produce MP4 recordings with what looks like the same kind of visual issue. And my phone's Android version is 6.0.1. |
|
Oh! I never received a mail notification for this issue, I didn't see it.
My first guess would be your encoder producing out-of-order frames (B-frames), not correctly managed by scrcpy. Since we want the lowest possible latency, we want frames in order anyway. It seems they added things related to frames out-of-order recently, and Android O and P briefly enabled B-frames: However:
What about: ?
Yes, please 👍 |
|
I think when I first posted the issue my github account was flagged, since I hadn't used it in several years, so it probably didn't notify for it. video.mp4 I most recently recorded this file using server code that I built locally with additional logging statements. It has fewer noticeable frame jumps, but the 8th frame or so seems to have a jump back. I've attached log files which are from the recording of this file too. file.mp4 My older file.mp4 serverLog.txt The PTS were ascending on the server, I logged the list of buffer flags and the hashCode of the codecBuffer before using it to write its contents to the socket. The hashCode values of each packet was unique, so it seems like it's not a duplication of a previous frame. When I do: The resulting mp4 file looks very similar to the file.mp4 file shared above. |
|
I tried changing a few more things to see if anything would help. I tried MediaCodec.flush() after each MediaCodec.releaseOutputBuffer, that makes all the frames into keyframes, which makes the video more blurry, and does not prevent the frame jumps. I tried setting the ScreenEncoder.DEFAULT_FRAME_RATE to 30, which allowed me to go 10 seconds on the title screen without any issues, but then when I tried to play the game which does a lot more graphical stuff the issues came back. Setting the DEFAULT_FRAME_RATE to 30 still produced more than 30 frames in each second, it seemed to produce around 40 frames each second. I tried setting it to 15 fps, but it still had lots of graphical issues recording gameplay; the resulting video file from that 15 fps setting seems to produce a file with about 30 frames each second. I suspect one of the ways that the gameplay for this game becomes more demanding on the recording system is that it probably uses a very large number of layers to display things. So it might be that the problem becomes worse the more layers are in use by the display. I don't know if there are any options to mitigate that. |
|
This is probably a bug in your hardware encoder :( |
|
I have a similar issue on LineageOS 16 on my OnePlus 3. What's really strange however, is that it doesn't happen on the stock ROM. |
|
I tried a few more things that didn't help, and then I thought to try having the server use a different codec to encode the video. I had my phone list out all of its encoders and the mimeTypes they support by adding MediaCodecList codecList = new MediaCodecList(MediaCodecList.ALL_CODECS);
MediaCodecInfo[] infoList = codecList.getCodecInfos();
for (int i = 0; i < infoList.length; i++) {
MediaCodecInfo codecInfo = infoList[i];
if (!codecInfo.isEncoder()) {
continue;
}
Ln.i(codecInfo.getName());
String[] types = codecInfo.getSupportedTypes();
for (int j = 0; j < types.length; j++) {
Ln.i("supports " + types[j]);
}And at first I wanted to try "OMX.google.vp8.encoder", since vp8 sounded like a different family of codec, but I couldn't figure out how to get the libav part on the client app to like what my phone was sending to it. That vp8 encoder was not sending a CodecConfig packet at the start, which seemed weird, and I stopped trying with that. But I noticed that my phone listed 2 encoders that both support mimeType "video/avc", they were named "OMX.qcom.video.encoder.avc" and "OMX.google.h264.encoder". The qcom encoder came up first when iterating, so I suspected when I used the unmodified scrcpy that the encoding was handled by that codec, and when I modified it to explicitly use that codec name for the encoder, I got the same results. When I change it to specify "OMX.google.h264.encoder", I no longer get the frame jumps. The video comes out quite well when I use that codec. I think it might be worth considering some kind of option to specify the codec, or something that lists the video/avc codecs on the phone and lets you pick. |
rom1v
changed the title
|
(Just a reminder) To list the encoders/decoders: |
|
There seem to be a lot of issues on this (#465, #1022, #1103), so I apologize if this is not the correct one to comment under. But hey, it's "666" and it's still open, so... how can I resist? I too am running into the IllegalStateException with some new android devices (running android 9) I have and it is resolved by using So for now I'll stick with the Here's what the error looks like on my laptop running scrcpy: Here's what the error looks like on the device (from logcat): Here's what the section from the media_codecs.xml looks like: |
I downloaded scrcpy 1.9 on 7/24/2019 with the goal of recording video from my Droid Turbo onto my Windows 7 x64 PC. When I connect with USB and run scrcpy and have it display to my screen, and the video from the phone has a lot of motion, it's common for the display from scrcpy to periodically appear to show maybe 1 frame that probably would have been the correct frame to display about 1/10 of a second ago. So whatever motion was happening appears to jump backwards to where it would have been maybe 1/10th of a second ago for just one frame and then jump back to what the phone is currently displaying; it's a very jarring appearance. Depending on the size I am displaying with scrcpy, these frame jumps may happen as frequently as once a second. If I reduce the display size very small like scrcpy --max-size 400, the frame jumps are less frequent, but they still happen occasionally, maybe a few times every 10 seconds.
I have had my phone display its CPU and memory usage while running this, and it seems to display around 15.00/14.00/14.00 as the CPU usage and the bar only seems to go around 15% across the screen so I interpret that as 15% CPU usage. The phone memory remains with over 1 GB free while this happens. My PC CPU usage is only around 5% while it happens, and only 4 out of 8 GB of memory are in use; these are when I'm running scrcpy --max-size 1920.
I also tried having scrcpy do the file recording. I ran one with mp4 and one with mkv, with the --no-display option so it wouldn't need to do double work, but the mp4 and mkv files it created still have the same kinds of frame jumps I would see in the live display. I play the files in VLC, and I've checked that I'm not just getting some kind of playback issue with the mp4 and mkv files by advancing individual frames in VLC and I can still see the motion get jumped backwards for a frame.
I do not usually run it with --render-expired-frames, and when I've tried it, the frame jumps seem just as frequent, possibly more frequent. When I use ctrl+i, I often see skipped frames, but I am fairly certain I have observed these frame jumps in the display even during periods where no skipped frames are logged. The display on the phone itself always looks perfect during the whole thing, and the game doesn't feel any less responsive than it would while scrcpy was not running.
Do you have any suggestions to get it where it doesn't end up displaying or recording an older frame after it has already displayed/recorded a newer frame? I can probably share the mp4 and mkv file with google drive or something later if that would help.
The text was updated successfully, but these errors were encountered: