Workaround for VLC not liking Pi MJPEG streams?

[6by9]

As has been noted before, VLC doesn't like the raw MJPEG stream that the Pi encoder produces. It detects the stream as J420 (YUV 4:2:0 with full range luma) and only displays one frame. Oddly it tends to get the duration about right.
MPlayer and avconv are happy with the files, as is VLC if you mux the data into a container.

Some interesting observations off waveform80/picamera#335 that adding a single space to the start of the file means VLC detects it as MJPEG. It's not practical to insert that as "header bytes" because they aren't, and how often would you repeat it for inline headers mode and splitting streams, and is it even a valid stream if you do that.
Using a USB webcam and V4L2 to capture an MJPEG stream it appears that the webcam doesn't insert the JFIF APP0 block whereas Pi does. It may be that removing it makes VLC happier, but not tested yet.

(The webcam also uses restart markers throughout the stream. I see little point for that, although the plumbing may be simple enough to do)

Raising this issue as a reminder to investigate what it is with VLC that upsets it. If it is easy to work around then we can do that, otherwise it is raise a bug on VLC.

[6by9]

If there is no workaround, then we may need to investigate adding container support to raspivid, although that still doesn't solve the problem for streams captured to file via V4L2.

[popcornmix]

VLC (or possibly libavcodec) does handle some surprising things like playing an .mp3 file that has been renamed to .wav.
I believe it tries the obvious codec (based on extension) and if that fails to parse, it falls back to a slower "probe" - trying each codec in turn and assigning a score for how well it matches.
Possibly the "insert space" hack causes it to go through the slower "probe" path which gets a better answer.
What extension do the mjpeg files have? Or is this being piped without a filename?

[6by9]

.mjpeg or .mjpg. Neither works.

Just looking at the ouptut from vlc -vvv, on a Pi generated stream we get

[0x7fd6ecc01788] image demux debug: Detected image: Motion JPEG Video
[0x138d008] dbus interface debug: Getting All properties
[0x7fd6ece59678] main image decoder debug: looking for decoder module matching "any": 39 candidates
[0x7fd6ece59678] avcodec image decoder debug: trying to use direct rendering
[0x7fd6ece59678] avcodec image decoder debug: allowing 4 thread(s) for decoding
[0x7fd6ece59678] avcodec image decoder debug: avcodec codec (Motion JPEG Video) started
[0x7fd6ece59678] main image decoder debug: using decoder module "avcodec"
[0x7fd6ece59678] avcodec image decoder debug: using direct rendering
[0x12e7638] qt4 interface debug: IM: Setting an input
[0x138d008] dbus interface debug: Getting All properties
[0x7fd6ece59678] main image decoder debug: removing module "avcodec"
[0x7fd6ece59678] avcodec image decoder debug: ffmpeg codec (Motion JPEG Video) stopped
[0x7fd724c845b8] main input debug: selecting program id=0
[0x7fd6ecc01788] main demux debug: using demux module "image"
[0x7fd724c845b8] main input debug: looking for a subtitle file in /home/dave/tmp/
[0x7fd6ecc73328] main decoder debug: looking for decoder module matching "any": 39 candidates
[0x7fd6ecc73328] main decoder debug: using decoder module "rawvideo"

vs with the space

[0x70c548] main playlist debug: processing request item: null, node: Playlist, skip: 0
[0x70c548] main playlist debug: rebuilding array of current - root Playlist
[0x70c548] main playlist debug: rebuild done - 1 items, index -1
[0x70c548] main playlist debug: starting playback of the new playlist item
[0x70c548] main playlist debug: resyncing on test.mjpeg
[0x70c548] main playlist debug: test.mjpeg is at 0
[0x70c548] main playlist debug: creating new input thread
[0x7fc5b80009b8] main input debug: Creating an input for 'test.mjpeg'
[0x7fc5b80009b8] main input debug: using timeshift granularity of 50 MiB, in path '/tmp'
[0x7fc5b80009b8] main input debug: `file:///home/dave/tmp/test.mjpeg' gives access `file' demux `' path `/home/dave/tmp/test.mjpeg'
[0x7fc5b80009b8] main input debug: creating demux: access='file' demux='' location='/home/dave/tmp/test.mjpeg' file='/home/dave/tmp/test.mjpeg'
[0x7fc5b0000e78] main demux debug: looking for access_demux module matching "file": 20 candidates
[0x7fc5b0000e78] main demux debug: no access_demux modules matched
[0x7fc5b80009b8] main input debug: creating access 'file' location='/home/dave/tmp/test.mjpeg', path='/home/dave/tmp/test.mjpeg'
[0x7fc5b00010c8] main access debug: looking for access module matching "file": 25 candidates
[0x7fc5b00010c8] filesystem access debug: opening file `/home/dave/tmp/test.mjpeg'
[0x7fc5b00010c8] main access debug: using access module "filesystem"
[0x7fc5b0001218] main stream debug: Using stream method for AStream*
[0x7fc5b0001218] main stream debug: starting pre-buffering
[0x7fc5b0001218] main stream debug: received first data after 0 ms
[0x7fc5b0001218] main stream debug: pre-buffering done 1024 bytes in 0s - 76923 KiB/s
[0x7fc5b0c01518] main stream debug: looking for stream_filter module matching "any": 9 candidates
[0x7fc5b0c01518] main stream debug: no stream_filter modules matched
[0x7fc5b0c01518] main stream debug: looking for stream_filter module matching "record": 9 candidates
[0x7fc5b0c01518] main stream debug: using stream_filter module "record"
[0x7fc5b80009b8] main input debug: creating demux: access='file' demux='' location='/home/dave/tmp/test.mjpeg' file='/home/dave/tmp/test.mjpeg'
[0x7fc5b0c01768] main demux debug: looking for demux module matching "any": 63 candidates
[0x7fc5b0c01768] mod demux debug: MOD validation failed (ext=mjpeg)
[0x7fc5b0c01768] ts demux debug: TS module discarded (lost sync)
[0x7fc5b0c01768] avformat demux debug: trying url: /home/dave/tmp/test.mjpeg
[0x7fc5b0c01768] avformat demux debug: detected format: mjpeg
[0x728538] qt4 interface debug: IM: Setting an input
[mjpeg @ 0x7fc5b0c09ca0] Estimating duration from bitrate, this may be inaccurate
[0x7fc5b80009b8] main input debug: selecting program id=0
[0x7fc5b0c01768] avformat demux debug: adding es: video codec = MJPG (8)
[0x7fc5b0c01768] avformat demux debug: AVFormat supported stream
[0x7fc5b0c01768] avformat demux debug:     - format = mjpeg (raw MJPEG video)
[0x7fc5b0c01768] avformat demux debug:     - start time = -1
[0x7fc5b0c01768] avformat demux debug:     - duration = -1
[0x7fc5b0c01768] main demux debug: using demux module "avformat"
[0x7fc5b80009b8] main input debug: looking for a subtitle file in /home/dave/tmp/
[0x7fc5b0c0a5b8] main decoder debug: looking for decoder module matching "any": 39 candidates
[0x7fc5b0c0a5b8] avcodec decoder debug: trying to use direct rendering
[0x7fc5b0c0a5b8] avcodec decoder debug: allowing 4 thread(s) for decoding
[0x7fc5b0c0a5b8] avcodec decoder debug: avcodec codec (Motion JPEG Video) started
[0x7fc5b0c0a5b8] main decoder debug: using decoder module "avcodec"
[0x7fc5b0c01998] main demux meta debug: looking for meta reader module matching "any": 2 candidates
[0x7fc5b0c01998] lua demux meta debug: Trying Lua scripts in /home/dave/.local/share/vlc/lua/meta/reader
[0x7fc5b0c01998] lua demux meta debug: Trying Lua scripts in /usr/lib/vlc/lua/meta/reader
[0x7fc5b0c01998] lua demux meta debug: Trying Lua playlist script /usr/lib/vlc/lua/meta/reader/filename.luac
[0x7fc5b0c01998] lua demux meta debug: Trying Lua scripts in /usr/share/vlc/lua/meta/reader
[0x7fc5b0c01998] main demux meta debug: no meta reader modules matched
[0x7fc5b80009b8] main input debug: `file:///home/dave/tmp/test.mjpeg' successfully opened
[0x7fc5b80009b8] main input debug: Buffering 0%
[0x7fc5b80009b8] main input debug: Buffering 13%
[0x7fc5b80009b8] main input debug: Buffering 26%

So definitely the parsing is going down a different route there.

The stream from my USB webcam via V4L2 seems to be different again:

[0x7fcd20c01518] main stream debug: looking for stream_filter module matching "any": 9 candidates
[0x7fcd20c01518] main stream debug: no stream_filter modules matched
[0x7fcd20c01518] main stream debug: looking for stream_filter module matching "record": 9 candidates
[0x7fcd20c01518] main stream debug: using stream_filter module "record"
[0x7fcd280009b8] main input debug: creating demux: access='file' demux='' location='/home/dave/tmp/temp.mjpeg' file='/home/dave/tmp/temp.mjpeg'
[0x7fcd20c01788] main demux debug: looking for demux module matching "any": 63 candidates
[0x7fcd20c01788] mod demux debug: MOD validation failed (ext=mjpeg)
[0x7fcd20c01788] ts demux debug: TS module discarded (lost sync)
[0x7fcd20c01788] mjpeg demux debug: JPEG SOI marker detected
[0x7fcd280009b8] main input debug: selecting program id=0
[0x7fcd20c01788] main demux debug: using demux module "mjpeg"
[0x7fcd280009b8] main input debug: looking for a subtitle file in /home/dave/tmp/
[0x7fcd20c026d8] main decoder debug: looking for decoder module matching "any": 39 candidates
[0x7fcd20c026d8] avcodec decoder debug: trying to use direct rendering
[0x7fcd20c026d8] avcodec decoder debug: allowing 4 thread(s) for decoding
[0x7fcd20c026d8] avcodec decoder debug: avcodec codec (Motion JPEG Video) started
[0x7fcd20c026d8] main decoder debug: using decoder module "avcodec"

So the question seems to be why does it drop back to
"[0x7fd6ecc01788] main demux debug: using demux module "image""
with the Pi stream, and image appears to look for a single frame only.

[6by9]

PS Please don't get sidetracked into debugging this issue (unless you really want to). This was more a note to myself so it didn't get forgotten.

As we discussed yesterday, probably the more productive solution is to add support for containers into raspivid, although this leaves V4L2 and PiCamera a little out in the cold.
The other thing to check is if VLC does a better job if we choose MJPEG A from within the codec. That rejigs the headers slightly but should be in a compatible manner. MJPEG B is no longer a straight concatenation of JPEG encoded frames, so probably wants to be avoided.

[6by9]

So the VLC chosen demux can be overridden with a simple --demux=avformat or --demux=mjpeg in the command-line, and it is then happy to play the Pi file.
So it really is just that initial demuxer choice that is going wrong.

[popcornmix]

Is just using --demux=mjpeg on vlc a sufficient solution?

[6by9]

It fixes the issue, but means you can't just double-click on a file. Leave this open for now and I'll see if I can make VLC happy with our streams out of the box, but I won't spend too long on it.
--demux=avformat is probably preferred over =mjpegas it then makes the correct choice for most file formats rather than barfing on anything that isn't mjpeg.

[6by9]

I think I've found the difference.
https://github.com/videolan/vlc/blob/master/modules/demux/image.c#L392 IsJfif will succeed on our stream, but fail on the other one. We insert the APP0 JFIF header whilst they don't. Likewise the space will cause it to fail. On failure it drops back to libavformat which does give the right answer (as does forcing the demux to be libavformat).

So removing the JFIF header would be a fix, as would reordering the blocks (which I think MJPEG-A will do). I need to do some more reading to work out which is the most appropriate solution.

[Ruffio]

@6by9 what did your reading end up with?

[6by9]

I've not had a chance to find a better solution on what to put in the bitstream.
There is the workaround to get it working, so this is still a low priority.

[JamesH65]

@6by9 Thoughts?

This issue will be closed within 30 days unless further interactions are posted. If you wish this issue to remain open, please add a comment. A closed issue may be reopened if requested.

[6by9]

I have a firmware patch that does drop the JFIF header from the MJPEG frames. Merge request created.

This improves matters somewhat in that VLC now identifies it as MJPEG and tries to use the mjpeg demux module. However the mjpeg demux seems to be slightly broken and drops most frames on the floor as being delivered too late. Add --mjpeg-fps 30 (or other appropriate frame rate) to the command line and it's happy and plays, but that's still as annoying as adding --demux=avformat (which also still works flawlessly).

Capturing an MJPEG stream off my laptop's webcam and I get the same issue of not playing without specifying the frame rate, so I'm of the view that we merge the change that selects the MJPG demux, but then close this issue as it seems to be a VLC one and not ours.

[6by9]

Closing.
We now use the correct demux, but VLC still appears to be requiring more hand holding. As this is true for webcams too, it really needs fixing in VLC and not on the Pi.

[KansaiUser]

I tried the suggested solution. It did not work. MJPEG still shows as static