No keyframes shown on Skydio 2 recordings. v6.0.x

[ohhorob]

On the recent update of the Skydio 2 firmware, the video files recorded don't display keyframes in LosslessCut.

main.f2afc881.chunk.js:1 Version 3.19.0
main.f2afc881.chunk.js:1 Load Object
main.f2afc881.chunk.js:1 getFormatData /Volumes/Untitled/DCIM/100SKYDO/S1000022.MP4
main.f2afc881.chunk.js:1 ffprobe -of json -show_format -i '/Volumes/Untitled/DCIM/100SKYDO/S1000022.MP4'
main.f2afc881.chunk.js:1 formats mov,mp4,m4a,3gp,3g2,mj2
main.f2afc881.chunk.js:1 fileType detected format {"ext":"mp4","mime":"video/mp4"}
main.f2afc881.chunk.js:1 ffprobe -of json -show_entries stream -i '/Volumes/Untitled/DCIM/100SKYDO/S1000022.MP4'
main.f2afc881.chunk.js:1 ffprobe -v error -read_intervals '0%30' -show_packets -select_streams 0 -show_entries 'packet=pts_time,flags' -of json '/Volumes/Untitled/DCIM/100SKYDO/S1000022.MP4'
main.f2afc881.chunk.js:1 Saving /Volumes/Untitled/DCIM/100SKYDO/S1000022-llc-edl.csv
DevTools failed to parse SourceMap: file:///Applications/LosslessCut.app/Contents/Resources/app.asar/build/static/js/2.84de25ef.chunk.js.map
DevTools failed to parse SourceMap: file:///Applications/LosslessCut.app/Contents/Resources/app.asar/build/static/js/main.f2afc881.chunk.js.map
DevTools failed to parse SourceMap: file:///Applications/LosslessCut.app/Contents/Resources/app.asar/node_modules/human-signals/build/src/main.js.map
DevTools failed to parse SourceMap: file:///Applications/LosslessCut.app/Contents/Resources/app.asar/node_modules/human-signals/build/src/signals.js.map
DevTools failed to parse SourceMap: file:///Applications/LosslessCut.app/Contents/Resources/app.asar/node_modules/human-signals/build/src/core.js.map
DevTools failed to parse SourceMap: file:///Applications/LosslessCut.app/Contents/Resources/app.asar/node_modules/human-signals/build/src/realtime.js.map
DevTools failed to parse SourceMap: file:///Applications/LosslessCut.app/Contents/Resources/app.asar/node_modules/uri-js/dist/es5/uri.all.js.map
DevTools failed to parse SourceMap: file:///Applications/LosslessCut.app/Contents/Resources/app.asar/build/static/css/main.3276fdc4.chunk.css.map

small sample file - https://www.dropbox.com/s/ib2pwhsxyk41qnz/S1000024.MP4.zip?dl=0

[wolfd]

I might be able to provide some insight into this. We include a thumbnail in our video files, which ffmpeg by default (for some reason) recognizes as an MJPEG stream at index 0. The actual video content is at index 1.

However, the MJPEG stream (which is only a single frame long), has a disposition of AV_DISPOSITION_ATTACHED_PIC. I believe one way to ignore it on the command line is as so: https://video.stackexchange.com/a/22287

A better way, depending on how LosslessCut works, is to use the output of ffprobe -of json -show_entries stream (which I see called above). The output has a entry located at streams[i].disposition.attached_pic that can be used to filter out MP4 thumbnails.

It's a weird ffmpeg behavior, but technically isn't a bug. This will happen in any MP4 that includes a thumbnail AFAIK.

For reference, this is the output of Ubuntu 18's ffprobe on that file on my machine.
ffprobe -of json -show_entries stream '/home/danny/Downloads/S1000024.MP4' 2>/dev/null

{
    "programs": [

    ],
    "streams": [
        {
            "index": 0,
            "codec_name": "mjpeg",
            "codec_long_name": "Motion JPEG",
            "codec_type": "video",
            "codec_time_base": "0/1",
            "codec_tag_string": "[0][0][0][0]",
            "codec_tag": "0x0000",
            "width": 384,
            "height": 216,
            "coded_width": 384,
            "coded_height": 216,
            "has_b_frames": 0,
            "sample_aspect_ratio": "1:1",
            "display_aspect_ratio": "16:9",
            "pix_fmt": "yuvj420p",
            "level": -99,
            "color_range": "pc",
            "color_space": "bt470bg",
            "chroma_location": "center",
            "refs": 1,
            "r_frame_rate": "90000/1",
            "avg_frame_rate": "0/0",
            "time_base": "1/90000",
            "start_pts": 0,
            "start_time": "0.000000",
            "duration_ts": 281970,
            "duration": "3.133000",
            "bits_per_raw_sample": "8",
            "disposition": {
                "default": 0,
                "dub": 0,
                "original": 0,
                "comment": 0,
                "lyrics": 0,
                "karaoke": 0,
                "forced": 0,
                "hearing_impaired": 0,
                "visual_impaired": 0,
                "clean_effects": 0,
                "attached_pic": 1,
                "timed_thumbnails": 0
            }
        },
        {
            "index": 1,
            "codec_name": "h264",
            "codec_long_name": "H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10",
            "profile": "High",
            "codec_type": "video",
            "codec_time_base": "140983/8460000",
            "codec_tag_string": "avc1",
            "codec_tag": "0x31637661",
            "width": 3840,
            "height": 2160,
            "coded_width": 3840,
            "coded_height": 2160,
            "has_b_frames": 0,
            "sample_aspect_ratio": "1:1",
            "display_aspect_ratio": "16:9",
            "pix_fmt": "yuvj420p",
            "level": 51,
            "color_range": "pc",
            "color_space": "bt709",
            "color_transfer": "bt709",
            "color_primaries": "bt709",
            "chroma_location": "left",
            "refs": 1,
            "is_avc": "true",
            "nal_length_size": "4",
            "r_frame_rate": "30/1",
            "avg_frame_rate": "4230000/140983",
            "time_base": "1/90000",
            "start_pts": 0,
            "start_time": "0.000000",
            "duration_ts": 281966,
            "duration": "3.132956",
            "bit_rate": "95239887",
            "bits_per_raw_sample": "8",
            "nb_frames": "94",
            "disposition": {
                "default": 1,
                "dub": 0,
                "original": 0,
                "comment": 0,
                "lyrics": 0,
                "karaoke": 0,
                "forced": 0,
                "hearing_impaired": 0,
                "visual_impaired": 0,
                "clean_effects": 0,
                "attached_pic": 0,
                "timed_thumbnails": 0
            },
            "tags": {
                "creation_time": "2020-04-12T16:46:46.000000Z",
                "language": "eng",
                "handler_name": "VideoHandle"
            }
        }
    ]
}

[dngrlux]

For anyone following along, I corrected all of my recent Skydio 2 footage with:
ffmpeg -i inputfile.mp4 -map 0:1 -ignore_unknown -copy_unknown -c copy outputfile.mp4

It then works as intended within LosslessCut, keyframes and all.

[mifi]

@wolfd thanks for digging! I think this will be a problem with any video file that has the thumbnail as video stream #0 (which may be all?) I'm already running ffprobe so I can filter out.

[wolfd]

Happy to help!

There may be a way for us get the thumbnail to show as stream 1, which we're investigating (some careful reordering of atoms in the MP4), but it may cause problems in other programs. MP4 parsers are weird, and have different expectations.

I still think it's still a good idea to filter out attached_pic streams, but it's possible that not all MP4s with thumbnails will have them as stream 0.

[mifi]

Yea I just implemented and tested. will be fixed in the next version!

[dngrlux]

I just tested with the sample footage above as well as my own Skydio 2 footage. The fix for showing keyframes is working, but the resulting exported video output only works in roughly half of my cuts. Many of the exported segments appear as solid black video in QuickTime on macOS. I could reproduce this with the sample footage above when making multiple cuts.

If I check the resulting segments of my own footage with ffprobe -loglevel error -skip_frame nokey -show_entries frame=pkt_pts_time -of csv=print_section=0 filename.mp4 the keyframes start at non-zero locations. e.g. 187.432000, 187.698578, 187.965156, 188.231844, 188.498422. Normally those would be 0.000000, 0.266667, 0.533333, 0.799878, 1.066500. Not sure if that is helpful or a non-issue.

If you need a longer sample video clip for testing, I can assist.

Update: I opened one of the problematic source videos, turned off stream 0 (MJPEG thumbnail), and exported the full video. I then re-opened that resulting video in LLC, marked numerous cuts and exported them. All of the segments play properly and have keyframes starting at 0.000000. There is something about the initial process above that is breaking the video clips that is resolved when processing them via this second method (open, export, open, proceed as normal).

Update 2: Turning off stream 0 and exporting segments works, too. If stream 0 is not manually turned off, much of the resulting video segments are unplayable.

[mifi]

Ohh awesome job researching this. Maybe I will change it so thumbnails stream defaults to off.

[ohhorob]

Just a quick note to say the latest Skydio 2 firmware has put the thumbnail track in the second position, and LosslessCut is reading and displaying the keyframes as expected!

[mifi]

Ok. But i still think there will be the issues when cutting if there is a thumbnail track. Therefore in the newest lc version i have now disabled thimnail tracks by default (But user can of course turn it on)