Best approach to "mix" a video playlist (MP4 with audio), with an audio playlist (Wav) and output a video stream (with both audios)

[kingpalethe]

Thanks for this great library!

I've spent hours with to documentation but I still have a lot of doubt about how to reliably implement this scenario.

I have a playlist of WAV files:

music_playlist = mksafe(playlist("audio-playlist.txt", mode="normal", reload_mode="watch"))

Each WAV file in the playlist currently looks like this in ffmpeg:

Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, 2 channels, s16, 1411 kb/s

---

I have a playlist of .MP4 files:

video_playlist = mksafe(playlist("video-playlist.txt", mode="normal", reload_mode="watch"))

Each MP4 in the playlist currently looks like this in ffmpeg:

Stream #0:0[0x1](und): 
Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 1024x1024, 2036 kb/s, 25 fps, 25 tbr, 12800 tbn (default)
Stream #0:1[0x2](und): 
Audio: aac (LC) (mp4a / 0x6134706D), 24000 Hz, mono, fltp, 81 kb/s (default)

You can see that these MP4 files have two streams -- a Video stream 0:0, and an Audio stream 0:1.
You can see that these WAV files have one stream -- and Audio stream 0:0.

Note -- if necessary, I could easily change the format of any of these streams, if that would make my goal possible or reliable in liquidsoap....

What I want to do is "mix" these playlists... To be as explicit as possible... I want to output a final Video stream which has

video channel:  the video from the .mp4
stereo output audio channel:   the mono audio from the .mp4, **overlayed/mixed** with the stereo WAV sound files

There are lots of examples in the documentation of various kinds of source.mux operations... For example, on page 151 of the manual, we find this example:

radio = playlist("~/Music")
video = single("image.jpg")
radio = source.mux.video(video=video, radio)
ykey = string.trim(file.contents("youtube-key"))
url = "rtmp://a.rtmp.youtube.com/live2/#{ykey}"
enc = %ffmpeg(format="flv",
%video(codec="libx264", pixel_format="yuv420p",
b="300k", preset="superfast", r=25, g=50),
%audio(codec="libmp3lame", samplerate=44100, b="160k"))
output.url(fallible=true, url=url, enc, radio)

But I think this is the wrong path for me, because I actually want to "mix" stream 0:1 of the video (i.e, the audio track of the video), with the WAV file.... and then output a new video stream.

And here is the tricky part -- the synchronization is really critical here, because these are videos of people speaking.. so even a second of bad synchronization will look bad!

Can anyone point me toward the right strategy to do this?

Clearly just using .add will not work -- liquidsoap seems to refuse to add an audio source to a video source.

I also see that there is a very new API, here https://www.liquidsoap.info/doc-dev/multitrack.html ... but I don't even find the word "mix" anywhere on this page!

This seems like a simple and common thing to do, but I'm struggling to figure out the right strategy.

One more thing -- if necessary, instead of using MP4 files as input (with and audio stream and a video stream), it would be no problem for me to pre-process these files, to split the audio and the video, and bring those two parts separately into liquidsoap, but this then brings up further doubts in my mind -- then they would have to be separate playlists, and for sure I would not get millisecond-tight synchronization.

And also: I know from experience that A/V sync issues are sometimes very tricky -- and have looked at this comment with some fear: #1091 (comment) ......

.... thanks again!

[kingpalethe]

Thinking about this more, I realized I should demo this with FFMPEG.

These three commands successfully deliver an export.mp4 which has the "mixed" audio:

ffmpeg -i video.mp4 -vn -c:a copy video_audio.aac
ffmpeg -i sound.wav -vn -c:a aac sound_audio.aac
ffmpeg -i video_audio.aac -i sound_audio.aac -filter_complex amix=inputs=2:duration=first:dropout_transition=2 output_audio.aac
ffmpeg -i video.mp4 -i output_audio.aac -c:v copy -c:a aac -map 0:v:0 -map 1:a:0 export.mp4

The magic seems to happen here: -filter_complex amix=inputs=2:duration=first:dropout_transition=2 -- using ffmpeg's amix filter.

So. I am now thinking that what I should be trying to do in Liquidsoap is use this same filter on my source playlists?

I see page 221, in the Liquidsoap manual:

FILTERS AND EFFECTS
def myfilter(s) =
  def mkfilter(graph) =
      a = ffmpeg.filter.audio.input(graph, source.tracks(s).audio)
      a = ffmpeg.filter.flanger(graph, a)
      a = ffmpeg.filter.audio.output(graph, a)
      v = ffmpeg.filter.video.input(graph, source.tracks(s).video)
      v = ffmpeg.filter.hflip(graph, v)
      v = ffmpeg.filter.negate(graph, v)
      v = ffmpeg.filter.video.output(graph, v)
      s = source.mux.track.audio(audio=a, s)
      s = source.mux.track.video(video=v, s)
      s
  end
ffmpeg.filter.create(mkfilter)
end
s = single("lf.mp4")
s = myfilter(s)

Maybe I need to adapt this, change input and output logic, and use the amix filter instead of hflip and etc....

[kingpalethe]

This is either very challenging, or maybe I am taking the wrong approach, but this is what I have so far:

video_sound = single("./input_test_video/video_audio.aac")
music_sound = single("./input_test_audio/music_audio.aac")


def myfilter(sound_one, sound_two)
    def mkfilter(graph) = 
        audio_one = ffmpeg.filter.audio.input(graph, sound_one)
        audio_two = ffmpeg.filter.audio.input(graph, sound_two)
        mixed_audio = ffmpeg.filter.amix(graph, audio_one, audio_two, inputs=2)
        mixed_audio
    end

end

video_sound = ffmpeg.raw.encode.audio_video(%ffmpeg(%audio.raw), video_sound)
music_sound = fmpeg.raw.encode.audio_video(%ffmpeg(%audio.raw), music_sound)
mixed = myfilter(video_sound, music_sound)

This fails with the error:

At ./test-filters.liq, line 23, char 44-63:
video_sound = ffmpeg.raw.encode.audio_video(%ffmpeg(%audio.raw), video_sound)

Error 5: this value has type
  format(audio=_)
but it should be a subtype of the type of the value at ffmpeg.liq, line 65, char 77
  format(audio=_, video=_, _)

But there are so many questions in my mind.. It doesn't seem clear to me that liquidsoap even supports "complex" filters, or filters with more than one input?

[toots]

Hi @kingpalethe and welcome!

I think that the biggest challenge that you will find is indeed synchronization.

While we have a pretty flexible and new support for multitrack muxing and demuxing, I'm not sure if you can do the level of granularity that you are lookin for.

The best option for you as of now would be to implement your own protocol.

At the higher level, the protocol would be taking whatever tracks you want to mux, fetch them, prepare a muxed track using an external command line such as ffmpeg and returning the resulting file. This way, you would be sure that every single request is muxed exactly and synchronously as you want them.

To be more specific, let's imagine that each track is identified with an id and that the main playlist contains the following

sync_mux:123
sync_mux:234

(you could even use the prefix parameter to avoid sync_mux:)

Then your protocol declaration would look like:

protocol.add(
  temporary=true,
  static=true,
  "sync_mux",
  fun (~rlog=_, ~maxtime=_, id) -> begin
    main_file = (... download file ..)
    audio_file = (... download file ...)
    result = file.temp("result", ".mp4")
    process.run("ffmpeg -i #{process.quote(main_file)} #{process.quote(audio_file)} ... ")
    [result]
  end
)

We'll be keeping in mind your use-case. I created an issue to track it here: #3262

[kingpalethe]

Thanks!

Regarding synchronization. Actually my synchronization requirement is limited....

Given the inputs:

Video/Audio input: A1, V1
Audio Input: A2

I need an output

A1 + V1 + A2

BUT -- I only need A1 + V1 to maintain synchronization... it's not important that these are tightly synchronized with A2

Explained another way. Imagine it's a TV show, with dialog (speaking). I want to add a music "soundtrack" or "bed" to go underneath the video + dialog. The soundtrack isn't really tightly timed to the video, I just want music to generally play "in the background".

I'm not sure if this makes the whole thing "easier" or not.

It occurs to me that a simple strategy might be TWO simultaneous liquidsoap scripts... one that sequences the video + audio playlist, and one that sequences the audio-only playlist. Each script would output an RTMP stream, and then, on the same machine, I would run a continuous FFMPEG process as shown here: https://stackoverflow.com/questions/18618191/ffmpeg-merge-multiple-rtmp-stream-inputs-to-a-single-rtmp-output

ffmpeg -i rtmp://ip:1935/live/AudioOnlyStream -i rtmp://ip:1935/live/AudioAndVideoStream -filter_complex "[0:a][1:a]amix[a]" -map 0:v -map "[a]" -c:v copy -f flv rtmp://ip:1935/live/OutputStream

I'm going to try this next and report back my findings here.

[toots]

Haaa. This should be much easier then.

Something like this:

tv_show = playlist("/path/to/shows")
soundtrack = playlist("/path/to/shows")

# Extract audio from tv shows, keep the rest separately.
let {
  audio=tv_show_audio,
  ...tv_show_tracks
} = source.tracks(tv_show)

# Extract audio from soundtrack
let {
  audio = soundtrack_audio
} = source.tracks(soundtrack)

# Mix audio tracks
mixed_audio = track.audio.add([tv_show_audio, soundtrack_audio])

# Remux everything:
mixed_source = source({
  audio = mixed_audio,
  ...tv_show_tracks
})

[kingpalethe]

Ah, yes, this works, thanks much.