Resources required to not have "We must catchup X.XX seconds!"

[sean-perryman]

I started testing on a small VPS, and the catchup would take about 2 minutes or so before it rolled over and restarted the stream. I went up to a 4 core 8 GB VPS and it's certainly better, maybe 10 minutes before rolling over.

Is there a magic number where it won't constantly play catchup? This is a stream of a small webm file to an icecast2 server on the same VM.

Do I need to just throw more CPU at it? Or is there a video format that is better for streaming?

[sean-perryman]

Actually, maybe I spoke too soon. As I let it cycle through several times the "catchup" message is actually counting back down, so maybe it's just a matter of letting it get settled? Gives me more hope for throwing more resources to it and it being good to go.

Also noticed that the stream is no longer skipping, so that is definitely a good sign!

[vitoyucepi]

Hi @sean-perryman,
I think catchup may be related to insufficient CPU performance.
Typically, video encoding demands significantly more resources compared to audio.

You can test if your VPS has enough power by excluding icecast and liquidsoap from the encoding chain.
For example, you can measure the encoding speed using ffmpeg.
ffmpeg -i small.webm -c:v libx264 -preset ultrafast -b:v 1000k -c:a aac -b:a 128k 1.mp4
You'll see the maximum speed relative to real-time.

You can cheat by

1. Using video in lower resolution. 640x480 is fine too.
2. Encoding everything beforehand and stream using copy-codec.
3. Using not-so-popular or dated encoders.

It is also plausible that the CPU is not the bottleneck affecting the performance.

[sean-perryman]

All of my sources are in mp4, I was planning on handbraking them to webm. I would need to scale them down at that step, right?

I've left the stream running for an hour or more and the catchup metric goes up to maybe 40 seconds, then back down to 2-3 seconds, then back up. Seems to just be cycling, but the stream quality doesn't seem to be suffering so might be good.

[vitoyucepi]

Consider measuring different performance metrics to see if any of them are exhibiting inappropriate behavior. Check steal and iowait.

You can stream to Icecast without liquidsoap to see if it is the root cause.
ffmpeg -re -i video.mp4 -c:v libx264 -c:a aac -f mpegts icecast://source:[email protected]:8000/live
You have the ability to adjust certain settings, such as the preset and x264-params options.
Chapter 7.3 of the Liquidsoap book is focused on encoders.
It would be helpful for you to search for the ffmpeg streaming guide for popular streaming services.

[Moonbase59]

@vitoyucepi is very right: It is a problem of CPU power with video, and very careful encoding settings. I’ve done lots of experiments (originally planning to generate H.256 HLS video) and ended up using an MPEG-TS H.264 stream, encoding with "ultrafast" and tuning "zerolatency", all else failed, although my station only produces 5 audio and 2 720p25 video streams. Interestingly, Icecast can be used to stream the MPEG-TS streams, with almost zero overhead.

My inputs are MP4 and MKV videos and DJ studio cams, usually H.264, some H.265, usually 1080p, but with no more than 30 fps. This seems to be important—in my setup using a 60 fps input is totally impossible (running AzuraCast, using their API, and doing some logo and lower-third overlays). The machine is a 4 CPU, 4 GB RAM, 80 GB SSD VM. I usually don’t bother waiting hours for handbrake, just because I want to plug a new VJ loop in, or something.

Here’s the encoding that worked best, using least CPU, with all my comments left in:

# Icecast video stream shall get its own clock, for decoupling
videostream = mksafe(buffer(s))
clock.assign_new(id="videostream", [videostream])

# reducing bitrate below b="5000k" produces horrible results
# if we leave the "b=" off, this will generate about 4000-12000 kbit/s
# so max. 3 listeners when selfhosting on 100 MBits/s (~30 MBits/s upload)
# If limiting bitrate, use "maxrate" and "bufsize" together
# (bufsize = 2 x maxrate seems to work good)
# Setting the GOP (group-of-pictures) to twice the framerate helps some decoders,
# most notably KODI. KODI 20.1 on a FireTV 4K stick also sucks above 10000k bitrates.
# CRF default is 23.
# It is certainly best to stream to a high bandwidth relay!
# Resonable 720p25 encoding can be had at the cost of about 190% CPU (multi-threaded).
#output.icecast(%ffmpeg(format="mpegts",
  #%audio(codec="ac3",channel_coupling=0),
  #%video(codec="libx264",g="50",
         #maxrate="8000k",bufsize="16000k",
         #preset="ultrafast", tune="zerolatency")),
         #id="videostream", host = "127.0.0.1", port = 8000, password = "myssecretpassword", mount = "/video.m2t", name = "Nite Radio", description = "Non-public test & evaluation server only", genre = "Various", url = "https://example.com/public/yourstation", public = false, encoding = "UTF-8", send_icy_metadata=true, videostream)

# AAC instead of AC3 -- compatible with more clients, notably mpegts.js
output.icecast(%ffmpeg(format="mpegts",
  #%audio(codec="ac3",channel_coupling=0),
  %audio(
    codec="aac",
    samplerate=44100,
    channels=2,
    b="192k",
    profile="aac_low"
  ),
  %video(codec="libx264",g="50",
    maxrate="8000k",bufsize="16000k",
    preset="ultrafast", tune="zerolatency")),
    id="videostream", host = "127.0.0.1", port = 8000, password = "mysecretpassword", mount = "/video.m2t", name = "Nite Radio", description = "Non-public test & evaluation server only", genre = "Various", url = "https://example.com/public/yourstation", public = false, encoding = "UTF-8", send_icy_metadata=true, videostream)

Hope this brings you a little further! (I’ve changed password, URL, and public=false.)