Combining ClippingMediaSource and ConcatenatingMediaSource

[bigahega]

Hello again,

so I have n videos and 1 audio file that I am trying to play as a single media. First I tried concatenating these n videos using the ConcatenatingMediaSource and then merge that concatenated media source with the audio file using MergingMediaSource; however, apparently this is not a possible option due to bunch of internal stuff going on there.

Then I tried to clip the audio file into n clips accordingly and then concatenate them into a single media source that is only audio, after that I tried merging them with the other concatenated media source of n videos. However the sound is restarting from t=0 every time the concatenated media source of videos switch to the next video.

Here is the first sample code I have tried for that:

fun SomeObject.magic(soundMediaSource: MediaSource): MediaSource {
    val mediaSource = ConcatenatingMediaSource(true)

    var lastTimeStamp = 0L
    videos.forEach {
        val durationUs = it.duration * 1000L
        mediaSource.addMediaSource(ClippingMediaSource(soundMediaSource, lastTimeStamp, lastTimeStamp + durationUs))
        lastTimeStamp += durationUs
    }

    return mediaSource
}

The problem with this code is that, when I merge this ConcatenatingMediaSource with the other ConcatenatingMediaSource containing n videos, the first clipped audio has a weird end position and does not end when the video ends so the audio continues but the first video of n videos just stops and waits for the sound to end, which takes forever.

The second code below, does not have such problem but instead of continuing from the last timestamp, the audio just restarts from the beginning:

fun SomeObject.fixedMagic(soundMediaSource: MediaSource): MediaSource {
    val mediaSource = ConcatenatingMediaSource(true)

    var lastTimeStamp = 0L
    videos.forEach {
        val durationUs = it.duration * 1000L
        mediaSource.addMediaSource(ClippingMediaSource(soundMediaSource, lastTimeStamp, durationUs))
        lastTimeStamp += durationUs
    }

    return mediaSource
}

How should I calculate the start position for the clipped audio tracks? Thanks in advance.

[bigahega]

A little bit more insight about the issue:

Here is how we are using the fixedMagic():

val concatenatedMediaSource = someObject?.otherMagic(context)
val soundMediaSource = ExtractorMediaSource.Factory(DefaultDataSourceFactory(context, Util.getUserAgent(context, "someAppName"))).createMediaSource(Uri.fromFile(audioFile))
val concatenatedSoundMediaSource = someObject?.fixedMagic(context, soundMediaSource)
val mergingMediaSource = MergingMediaSource(concatenatedMediaSource, concatenatedSoundMediaSource)
player.prepare(concatenatedSoundMediaSource)

and here is the otherMagic() implementation:

fun SomeObject.otherMagic(context: Context?): ConcatenatingMediaSource {
    val dataSourceFactory = DefaultDataSourceFactory(context, Util.getUserAgent(context, "someAppName"))
    val mediaSource = ConcatenatingMediaSource(true)
    videos.forEach {
        val tmpMediaSource = ExtractorMediaSource.Factory(dataSourceFactory).createMediaSource(Uri.fromFile(it.videoFile))
        mediaSource.addMediaSource(tmpMediaSource)
    }
    return mediaSource
}

Now check here:

val mergingMediaSource = MergingMediaSource(concatenatedMediaSource, concatenatedSoundMediaSource)

If I give the videos first into the MergingMediaSource, the videos are fine but the audio restarts every time the video switches to the next; however if I give the audio first then the videos as arguments into the MergingMediaSource, audios are playing fine, just like I wanted to clip them, but the videos are bugged.

[AquilesCanta]

Is it possible that the audio file you are clipping is not seekable?

[AquilesCanta]

Please provide the media so we can have a look.

[bigahega]

Yes the audio file is definitely seekable, and I am not referring to a single audio file here. I have tried a good many different audio files and they are all seekable. For instance the following code works perfectly for the every audio file I have tried: (the player has the audio as ExtractorMediaSource in the following code)

fun instantSeek(timeMs: Int) {
    player?.seekTo(timeMs.toLong())
}

[bigahega]

Could you look into the issue that I have pointed out? @AquilesCanta

[ojw28]

@tonihei - I reckon this was probably broken by [Internal CL ref cr/192621719]. Although perhaps it didn't work before that, either :). Behavior of ClippingMediaPeriod.ClippingSampleStream.readData and MergingMediaPeriod.readDiscontinuity are probably both worth looking at for the case described.

[bigahega]

Is there any workaround in your mind so that I can get this to work somehow? Basically I want to play a single audio file from 0 to t secs and N video files with a total duration of t secs.

[tonihei]

Can't reproduce with the latest dev branch, but it may have been fixed by this commit. Can you try the the dev-v2 branch to see if it still occurs? The underlying reason is probably a duplicate of #4873.

[bigahega]

Tried the dev-v2 branch through Jitpack within our main project and I'm sure I'm not missing something out but the project doesn't compile and yields a good many Type mismatch errors during compile time regarding Player!-ExoPlayer.

I will try it in a fresh project when I have some more time, hope it is indeed fixed.

[tonihei]

Did you have a chance to verify it's working now? We'll also publish a 2.9.1. soon containing the fix above.

[tonihei]

Closing due to inactivity and because we received no further info.