Long-term sync issues between audio and video in %ffmpeg

[dhannyz]

@toots
The stream comes up, plays ok, but a little glitchy, and then sometimes liquidsoap crashes with

2020/02/14 06:12:47 [clock.main:2] Source output(dot)url failed while streaming: FFmpeg.Avutils.Error(Input/output error)!
2020/02/14 06:12:47 [clock.main:3] Raised at file "avutil/avutil.ml", line 69, characters 51-77
2020/02/14 06:12:47 [clock.main:3] Called from file "av/av.ml", line 256, characters 20-60
2020/02/14 06:12:47 [clock.main:3] Called from file "av/av.ml", line 256, characters 20-60
2020/02/14 06:12:47 [clock.main:3] Called from file "encoder/ffmpeg_encoder.ml", line 212, characters 22-62
2020/02/14 06:12:47 [clock.main:3] Called from file "encoder/ffmpeg_encoder.ml", line 228, characters 15-22
2020/02/14 06:12:47 [clock.main:3] Called from file "outputs/output.ml", line 258, characters 24-44
2020/02/14 06:12:47 [clock.main:3] Called from file "outputs/output.ml", line 186, characters 40-56
2020/02/14 06:12:47 [clock.main:3] Called from file "clock.ml", line 288, characters 14-22

The output section of my script is as follows

encoder = %ffmpeg(
    video_codec="libx264",
    %video(b="4M", preset="ultrafast"),
    audio_codec="libfdk_aac",
    %audio(b="128k",afterburner=1),
    g=30,
    format="flv"
)

s = buffer(fallible=true, buffer=5., max=10., s)

output.url(fallible=true, url="rtmp://a.rtmp.youtube.com/live2/<key>",encoder,s)

You can see I tried adding a buffer to smooth things out a bit

Without the buffer it seems sometimes it doesnt connect at all and hangs.

Also some questions about missing ffmpeg settings. there does not seem to be an -r=30 (framerate) equivalent, however i assumed setting set("frame.video.samplerate",30) may account for this?
also -bsf (bitstream filter) and -pix_fmt seem to be missing?

Also it seems the resolution defined in the liquidsoap set() frame options is not preserved through the ffmpeg encoder, how should this be defined?

Also having some issues where the process hangs when trying to end it, and I'm forced to SIGKILL it

• Z

[Allavaz]

Why aren't you using output.youtube.live?

[toots]

@Allavaz The new %ffmpeg support should eventually be a much better alternative. We're still working on it.

@dhannyz Thanks for testing and for the report. I'm seeing some issues now too on my end, I'll investigate as soon as possible.

[toots]

pix_fmt should be available via integer arguments and possibly string. Framerate is the global frame rate for now but I'll add an encoder-specific fps indeed. bdf I'm not sure. How do you use it in ffmpeg normally?

[toots]

@dhannyz I've just added framerate support to %ffmpeg. Also, shipped a bugfix in ocaml-ffmpeg, make sure you get the latest one!

It seems that I can reproduce the freeze sometimes, will investigate next.

[dhannyz]

@toots great stuff, still need clarification on how output resolution is configured.
My YT streams were only seen at 360p despite being configured at 720p.

Am also looking at the ffmpeg.filter stuff, looks great. One question I have is for filters like ffmpeg.filter.asplit and ffmpeg.filter.hstack - how are multiple sources retrieved from and supplied to filters ?

@Allavaz short answer is I'm doing some pretty complex stuff currently with output.external and af fmpeg command that generates a visualizer and streams to many endpoints. Future ideas are even more complex

[smimram]

Small remark: should we change the default implementation of output.youtube.live to this?

[toots]

@smimram Sure, when it's proven stable enough.

@dhannyz Just added support for changing width/height in ffmpeg encoder. You should now be able to set a large value globally and encode to different sizes if needed.

[dhannyz]

@toots thanks for those changes. does definitely seem more stable. Liquidsoap has stopped crashing, and I can CTRL+C out of it without fail (yet)

However im having a major issue with huge A/V desync. audio seems to be upward of 5 to 10 seconds ahead of video.

I've been eliminating sections of the script that I thought may be the cause, like muxing default stock videos with audio and rotating between that and real video files. However I now have only a pure video playlist and still have the extremely large desync.

I also had youtube complain about the keyframe interval (which might be a cause of the desyncs?) there doesn't seem to be a way to set this. For example there is no keyint=GOPSIZE, min-keyint=GOPSIZE options available when using libx264

The general g=GOPSIZE argument seemed not to do anything

[toots]

Thanks for the feedback again. I have just added a change that makes sure the encoded audio stream has a pts set just like video, that could explain the temporal drift. Would you mind testing again?

Have you tried setting those options using the %video selector:

%ffmpeg(
  video_codec="libx264",
  %video(keyint=..., min-keyint=...),
  ...
)

As for the filters with dynamic input or output such as ffmpeg.filter.asplit, I haven't figured out we're gonna use them so for now, they're not really usable. I think the plan for complex filters like that would be to support ffmpeg's graph parsing syntax but don't quote me on that.

[autonarcosis]

I don't know how many times I've reported A/V sync issues. In the past 5 years I've never cracked it. The only way to not have A/V sync issues it to use the built in theora/vorbis codec. Not even sure that works anymore.

The one consistent thing I have found is that all the source files must have the same frame rate. Mixing 24fps and 30fps will immediately cause A/V sync issues on the transition from one source to another. I can't find my old configurations, from what I can remember I had to use +igndts and +genpts. There's also on the audio side first_pts=0. Dropped and Duplicating frames can also cause sync issues as well.

Setting the GOP and keyint is critical for DASH/HLS and must be a multiple of the frame rate. Not some random number. If your output is 25fps and you want GOP every 6 seconds the GOP and keyint should be 150.

I would make sure that the actual internal raw AVI video is synchronized or not. Dump the raw output and see if it is de-synchronized or not. If it is, this is an uphill battle which you won't ever win. I suspect this is the ultimate problem. There's just been something inherently wrong with the internal AVI generation for years.

And of course as I said, all of this never worked properly so I ended up giving up. I give it a try about once a year. What does work properly is jb-alvarado's ffplayout, which is essentially a python script that creates a continuous mpegts using piping methods and several instances of ffmpeg. I believe I did create a possible working scenario using LS and GStreamer output using mpegts via UDP. Again, can't stress this enough, all source material cannot change fps.

Welcome to hell.

[toots]

@autonarcosis, this issue is about the internal ffmpeg encoder. A lot of the trouble you're describing should go away once this encoder is stable. Any chance you could try it yourself as well?

[autonarcosis]

@toots
I would love to but no amount of git, opam commands I try can I get 1.5.0 to compile. I've always gotten the;

No Solution Is Found ....

[toots]

@autonarcosis Wanna open another issue describing how you installed and what your issue/error logs are? Happy to look at it..

[autonarcosis]

@toots
Sad to say. Audio de-syncs. Will take awhile to figure out at what point it occurs. It seemed stable for at least the first 6 or so files. All source files are various files from archive.org re-encoded in Theora/Vorbis 25fps. Side notes; I put the aresample in this config because at the time the audio resample configuration for Liquidsoap was not functioning. ffmpeg no longer works with icecast using webm (vp8/vorbis) due to EBML header non-sense (will require re-compiling Icecast Server). VP8/Vorbis was too slow even on older dual Xeon. Went with h264 DASH. I'll tweak the settings tomorrow, specifically the vsync and audio filter.

output.external(%avi,"ffmpeg -loglevel quiet -f avi -i pipe:0 -map 0:0 -c:v libx264 -preset veryfast -profile:v baseline -level:v 3. 0 -sc_threshold 0 -x264-params scenecut=0:open_gop=0:min-keyint=150:keyint=150 -filter:v format=pix_fmts=yuv420p -vsync 1 -r 25 -min rate 512k -maxrate 640k -bufsize 1280k -b:v 512k -map 0:1 -c:a aac -b:a 64k -filter:a aresample=48000:async=1:min_hard_comp=0.100000 :first_pts=0 -f dash -adaptation_sets 'id=0,streams=v id=1,streams=a' -dash_segment_type mp4 -window_size 6 -use_timeline 0 -use_tem plate 1 -extra_window_size 1 -seg_duration 6 -remove_at_exit 1 -index_correction 1 -utc_timing_url 'https://www.monkeykitty.com/date .php' -init_seg_name 'init-stream$RepresentationID$.m4s' -media_seg_name 'chunk-stream$RepresentationID$-$Number%05d$.m4s' -streamin g 1 /home/geoff/public_html/test_live_manifest.mpd",mksafe(videoplaylist))

[toots]

@autonarcosis It looks like you are still using the external ffmpeg encoder. The doc for the internal %ffmpeg encoder is here: https://www.liquidsoap.info/doc-dev/encoding_formats.html

@dhannyz and other people interested in this encoder: I just pushed some changes to the syntax, now it looks like:

%ffmpeg(format=<format>,
        # Audio section
        %audio(codec=<codec>,<option_name>=<option_value>,..),
        # Video section
        %video(codec=<codec>,<option_name>=<option_value>,..),
        # Generic options
        <option_name>=<option_value>,..)

Example:

%ffmpeg(format="mpegts",
        %audio(codec="ac3",channel_coupling=0),
        %video(codec="libx264",b="2600k",preset="ultrafast"))

That seems much more clean and logical to me..

[dhannyz]

@toots yes that makes a lot more sense now. Unfortunately I'm still getting extremely bad issues with sync, it looks like maybe liquidsoap is reading video in too fast if the framerates are different. eg: source file at 25fps and liquidsoap internally at 30fps

On another note i seem to be having issues with some unicode characters when drawing text. I know the font supports the characters. At the moment I am testing by writing the metadata to a file that my external ffmpeg reads via the drawtext filter (I tried using the ffmpeg.filter.drawtext however I couldnt figure out how to update the text when metadata updates)
Interestingly writing to that file also seems to glitch on some unicode characters and not render properly at all - But they work fine when I manually insert that text into the file - so its not an ffmpeg issue

EDIT: Have just set a bunch of videos to all the same fps and matched with the liquidsoap internal setting - sync seems better so far, will leave it on a while to test

[autonarcosis]

@dhannyz

In my testing I found that the source files must match the video FPS and the audio sample rate. For example if Liquidsoap is configured for 25fps and 44.1khz audio, the source files must be that.

Frame rate mismatches is immediate on track change.

Audio sample rate takes time. My source files were 48khz and I was getting consistent gap errors of about 200 microseconds when Liquidsoap was configured for 44.1khz. Nearly imperceptible unless you are highly familiar with the source material. This would give the slight perception that the first few tracks seemed OK, but over hours that sync expands. After about 6 hours the de-sync is at about 4 seconds. Obviously noticeable. Once I changed Liquidsoap to 48khz to match my source files, there were no more gap errors and the stream maintained sync.

[toots]

Thanks that's very interesting. We're suppose to convert frame rate for video so I'll double check on that. Also, we've historically been doing 44.1kHz/25fps but it seems that the standard for film is 48kHz/24fps, which obviously lines up better. I'll have a look at that as well, it's possible that we're off by some samples. Be right back!

[toots]

@autonarcosis Are those testing info based on using the latest ffmpeg builtin encoder? And what decoder are you using for your input files?

[autonarcosis]

@toots
Yes, as of about last week or so? Liquidsoap 1.5.0+git@1c6c160
I am using ffmpeg as the internal decoder. The source files are theora/vorbis.

[toots]

Thanks! I'm gonna try to reproduce the issue next.

[dhannyz]

@toots Just from what i know about ffmpeg I've had the same issue when setting -r <framerate> on its own. I would then fix it with a fps filter. The symptoms here seem to be similar.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue was closed for lack of activity. If you believe that it is still relevant, please confirm that it applies to the latest released version of liquidsoap and re-open the ticket. Thanks!