Loading future DASH segments in v4.x

[boredom2]

Have you read the FAQ and checked for duplicate open issues?
yes - this is (probably) identical to (closed) Ticket 3717

What version of Shaka Player are you using?
3.3.1

Can you reproduce the issue with our latest release version?
yes

Can you reproduce the issue with the latest code from master?
didnt check

Are you using the demo app or your own custom app?
both

If custom app, can you reproduce the issue using our demo app?
yes

What browser and OS are you using?
Chrome/Edge/Firefox

For embedded devices (smart TVs, etc.), what model and firmware version are you using?
n/a

What are the manifest and license server URIs?

https://sdn-global-streaming-cache.3qsdn.com/24666/files/21/10/11/3960246/24666-GVQnP7CqzRgXNDW.ism/manifest.mpd

What configuration are you using? What is the output of player.getConfiguration()?

{"preferredAudioLanguage":"auto","abr":{"enabled":true,"switchInterval":6},"manifest":{"disableText":true},"streaming":{"smallGapLimit":1,"jumpLargeGaps":true,"ignoreTextStreamFailures":true,"useNativeHlsOnSafari":false}}

What did you do?

Loading the Manifest "https://sdn-global-streaming-cache.3qsdn.com/24666/files/21/10/11/3960246/24666-GVQnP7CqzRgXNDW.ism/manifest.mpd" to the Player

What did you expect to happen?
The Video should be played until the end and then stop

What actually happened?

I already had an own Ticket on this - and Ticket 3717 was dealing with this too (but is closed now). From my Understanding, this should have been fixed in 3.3.1 - but it clearly isnt.

SHAKA starts to load non-existing ("phantom") Chunks like:

https://sdn-global-streaming-cache.3qsdn.com/24666/files/21/10/11/3960246/24666-GVQnP7CqzRgXNDW.ism/dash/24666-GVQnP7CqzRgXNDW-audio=128000-150.m4s
https://sdn-global-streaming-cache.3qsdn.com/24666/files/21/10/11/3960246/24666-GVQnP7CqzRgXNDW.ism/dash/24666-GVQnP7CqzRgXNDW-video=2793000-150.m4s

which leads to 404 - for some reason, it never stops requesting them, what leads to a ton of issues in the CDN, as millions of requests are processed there for non-existing Material.

This is ONLY happening in DASH, not in HLS.
(compare the HLS Manifest: https://sdn-global-streaming-cache.3qsdn.com/24666/files/21/10/11/3960246/24666-GVQnP7CqzRgXNDW.ism/manifest.m3u8)

[haisk]

Got this problem too. The main reason is that your Dash manifest is using Segment Template, where segments created based on duration and timescale . For some reason, when combine with certain content it will generate "phantom" segment like that at the end of the content duration.

You can change your origin (unified streaming) setting to not use Segment Template (using SegmentList or byte-range for example) and the problem will gone.

[boredom2]

Thanks for that insight. Ill talk with the backend team, if thats possible. But im sure, Shaka will fix this soon 😉

[avelad]

Can you test with v4.3.3? Thanks!

[github-actions]

Closing due to inactivity. If this is still an issue for you or if you have further questions, the OP can ask shaka-bot to reopen it by including @shaka-bot reopen in a comment.

[boredom2]

@shaka-bot reopen

[boredom2]

Sorry, I did not see the last Comment on this before.
I sadly must confirm, that nothing changed - the given DASH URLs above still dont work, the same (non-existing) Chunks are trying to be loaded and the Player dies.

[cristian-atehortua]

Hi @avelad, looks like this is happening to us, too. We're using the latest Shaka version. I'll share a manifest that has the problem.

[Canta]

I'm affected by this problem too.

The ticket title says "not existing chunks", which is correct. But I'll like to add a precision to that: it's not just "not existing chunks", but "future chunks". I'm working with 24/7 live streams, and Shaka player tries to download yet not existing chunks: future chunks.

Since I didn't found anything weird in my setup (I've been using it for years), i began to suspect to some new shaka player version. Indeed, I wasn't able to reproduce the problem with a 3.x.x version, while it doesn't take long to appear with a 4.x.x version.

Below are two screenshots, both taken about the same time two hours ago. The URLs are redacted for privacy. But please note the chunks numbers and the shaka version: 3.x is on the "473" chunk position and playing fine, while 4.x is on "495" and failing with 404 errors.

In those screenshots, the error is: 4.x version should not jump to "495". Those chunks does not yet exist by that time.
I don't have the exact manifest for that time, but I have a fresher one, which has the same structure and just changes the actual chunks timings.

manifest.mpd.zip

This is a very troubling issue, as shaka player is part of many production setups, and the issue doesn't seem to be originated by any improper manifest syntax. Given that it's about future chunks, most likely shaka player would be doing something wrong on the "next chunk" calc logic.

[joeyparrish]

@Canta, thank you for your insights. We will work on this when we have time, but the team is very much overloaded these days. It may be best for someone else in the community to investigate and try to determine the root cause.

[Canta]

@joeyparrish , I've been doing a quick search on the code, but I'm also on a tight agenda, so can't do much about this right now. Here are some notes nonetheless:

1. The original problem and mine looks the same, but I now realize that @boredom2 manifest is VOD, while mine is LIVE. That's static vs dynamic, and they have different logics. So most likely this is not the same issue. For the time being, I see no problem in using this ticket to analyze both, but the original problem looks like a "end + 1" problem, while mine is a jump in time.

2. I can see in lib/media/presentation_timeline.js this code for getting live edge timestamps:

  getLiveEdge_() {
    goog.asserts.assert(this.presentationStartTime_ != null,
        'Cannot compute timeline live edge without start time');
    // Date.now() is in milliseconds, from which we compute "now" in seconds.
    const now = (Date.now() + this.clockOffset_) / 1000.0;
    return Math.max(
        0, now - this.maxSegmentDuration_ - this.presentationStartTime_);
  }

That means there's 3 variables involved: now, start time, and max segment duration. Given that both times don't change (my clock is fine, and manifest start time only changes after restarts that didn't happened), I suspect something may be happening with the max segment duration calculation.
If you take a look at the manifest I've uploaded before, you can see the video and subs chunks are all repeated with the same duration, but audio is not. That's because the chunk duration (6 seconds) is a multiplier of the video GOP times, but not a multiplier of the audio cutting times. And if you take a look at the screenshots, the rendition with trouble is the audio one being played. So perhaps max segment duration is doing something wrong, altering the live edge timestamp in a way the player jumps in time.
Another posibility of the same problem would be that clockOffset_ variable, affecting now. IDK where that offset comes from but, if it's from MPD@presentationTimeOffset, there isn't one in my manifest, so should be 0.
Those are points to take a look at.

3. Looking at git log ./lib/media/presentation_timeline.js, I saw this change: https://github.com/shaka-project/shaka-player/commit/ea6774a1fde3569df4a00992ed17093abefc73e9. Didn't take a deep look at it, but it talks about handling "future segments". It's from March 2023, which would explain why I didn't saw this problem years ago. This could be another debug line.

If I clear my agenda, I'll take a look at it. In the meantime, I'll let quick notes so others can follow. @lifofernandez

[lifofernandez]

Hi @joeyparrish and @avelad, I'll share some research I did on this issue.

When Segment Template generates information of the streams, It creates a TSI for each one, appends and evicts segments to this TSI but does not always update the startNumber with each downloaded manifest.
So I assumed that with an updated startNumber, according to the current manifest, the math to compose the segment number would be within the expected.

Given this I modified the TSI's appendTemplateInfo method so that if the startNumber of the incoming manifest is different from this.templateInfo_.startNumber, it is recorded in acctualStartNumber (a different variable so as not to introduce new bugs or spread the problem to highers levels of abstraction).

if (this.templateInfo_.startNumber !== info.startNumber) {
    console.log(`TSI appendTemplateInfo:
    currentStartNumber ${this.templateInfo_.startNumber}
    and newStartNumber ${info.startNumber} differs.`);
    this.acctualStartNumber_ = info.startNumber;
    console.log(`TSI acctualStartNumber_:`, this.acctualStartNumber_);
}

A part of this new actual acctualStartNumber in the calculation, It was also necessary to modify position contemplating the amount of evicted segments (correctedPosition) so that the number is the expected one.

const segmentReplacement = correctedPosition + this.acctualStartNumber_;

Future chunk numbers are no longer observed, but it did produce many Possible encoding problem detected! Unexpected buffered range for reference. By setting the variable timeReplacement in seconds instead of units (range.start instead of range.unescaledStart and scaledPresentationTimeOffset instead of unscaledPresentationTimeOffset) the warnings of encoding problems decreased considerably.

const timeReplacement = this.templateInfo_.scaledPresentationTimeOffset + range.start;

Although it is necessary to review and fine-tune all the variables involved to achieve the best possible reproduction, I hope these notes helps to find the cause and fix of the problem.