Stream interrupted playback with Live DASH Halo pakager using Shaka 2.3.0 version

[hdodev]

Have you read the FAQ and checked for duplicate open issues?:
Yes, it looks like item #838, and to problems related to "presentationTimeOffset" in general. But the error log I have in console is different, and version used is higher.

What version of Shaka Player are you using?: 2.3.0

Can you reproduce the issue with our latest release version?: YES

Can you reproduce the issue with the latest code from master?: YES

Are you using the demo app or your own custom app?: CUSTOM

If custom app, can you reproduce the issue using our demo app?: NO, HTTP_ERROR 1002, but I'm sure the manifest is accessible!

What browser and OS are you using?: CHROME on UBUNTU

What are the manifest and license server URIs?:
Clear DASH = http://195.36.159.232/cleardashSO_tml/france4/manifest.mpd

What did you do?
I try to playback a Live Stream from my packager. We've just corrected our manifest thanks to your analysis last year : The problem was the fact that presentationTimeOffset appeared to be modified when the manifest was updated.

What did you expect to happen?
Live playback works continuously. If you can please help us to understand the remaining problem. Thanks in advance for your help.

What actually happened?
The playback starts, and then pauses, and then starts again, and so on.
It can appear once, twice, or more, there is no rule. Here is the log with the debug version:
Refusing to rewrite original references on update!
segment_index.js:319 Assertion failed: SegmentReferences are incorrectshaka.media.SegmentIndex.assertCorrectReferences_ @ segment_index.js:319shaka.media.SegmentIndex.merge @ segment_index.js:203shaka.dash.SegmentTemplate.createStream @ segment_template.js:82shaka.dash.DashParser.parseRepresentation_ @ dash_parser.js:1092shaka.dash.DashParser.parseAdaptationSet_ @ dash_parser.js:991shaka.dash.DashParser.parsePeriod_ @ dash_parser.js:733shaka.dash.DashParser.parsePeriods_ @ dash_parser.js:634shaka.dash.DashParser.processManifest_ @ dash_parser.js:521xlinkOperation.promise.then @ dash_parser.js:432

[joeyparrish]

Here are some numbers from my analysis:

At time 1519673732 (seconds since 1970 UTC, the unix epoch), I fetched your manifest. At that moment, availabilityStartTime was 1970-01-01T00:00:00.000Z (same as the unix epoch) and the video segments were described as:

  <Period id="5074585002895" start="PT1519672808.560S">
    <AdaptationSet mimeType="video/mp4" ...>
      <SegmentTemplate
          timescale="90000"
          initialization="$RepresentationID$-Header-1519672026560.m4s"
          media="$RepresentationID$-0-T-$Time$.m4s"
          presentationTimeOffset="5074655382895">
        <SegmentTimeline>
          <S t="5074655382895" d="180000" r="449"/>
        </SegmentTimeline>

The live edge is now, which is 1519673732 seconds since availabilityStartTime. So the most recent segment should have roughly this timestamp in seconds.

The first segment has timestamp 5074655382895 (in timescale units), and the last segment has start time 5074655382895 + 180000 * 449 = 5074736202895. We then subtract the presentationTimeOffset of 5074655382895 (also in timescale units) to arrive at 80820000 in timescale units.

Converting 80820000 from timescale of 90000 to seconds yields 898 seconds. We then add the period start time of 1519672808.560 seconds and get 1519673706.560 seconds.

How far is that from "now"? We subtract the time the request was made (1519673732) and get -25.44, so this is 25.44 seconds in the past.

So the timestamps are not that far off. Probably, you just need a suggestedPresentationDelay attribute to cover up this lag. A value of 30s might suffice.

To test this theory, I seeked backward by about 45 seconds and playback started.

It's not clear why playback didn't start without this action, though, and the JS console still spits out many warnings:

streaming_engine.js:1330 (video:1) cannot find segment: currentPeriod.startTime=1519673416.56 lookupTime=933.4669899940491

streaming_engine.js:1330 (audio:8) cannot find segment: currentPeriod.startTime=1519673416.56 lookupTime=933.4669899940491

segment_index.js:163 Refusing to rewrite original references on update!

segment_index.js:323 Assertion failed: SegmentReferences are incorrect

I'll dig deeper and let you know what I find.

[joeyparrish]

A side-note: in this case, HTTP_ERROR 1002 is caused by your manifest being an http URL and the demo app being an https URL. This is a "mixed-content" error, in which https origins can't load plain http resources.

[joeyparrish]

Although the timestamps all make sense in a single snapshot of your manifest, comparing over time reveals problems.

First snapshot:

  <Period id="5075233002895" start="PT1519679226.561S">
    ...
      <SegmentTemplate
        timescale="90000"
        initialization="$RepresentationID$-Header-1519679226561.m4s"
        media="$RepresentationID$-0-T-$Time$.m4s"
        presentationTimeOffset="5075233002895">
        <SegmentTimeline>
          <S t="5075233002895" d="180000" r="449"/>
        </SegmentTimeline>
      </SegmentTemplate>

Second snapshot, a couple seconds later:

  <Period id="5075233002895" start="PT1519679232.561S">
    ...
      <SegmentTemplate
         timescale="90000"
         initialization="$RepresentationID$-Header-1519679226561.m4s"
         media="$RepresentationID$-0-T-$Time$.m4s"
         presentationTimeOffset="5075233542895">
        <SegmentTimeline>
          <S t="5075233542895" d="180000" r="449"/>
        </SegmentTimeline>
      </SegmentTemplate>

The timestamps are changing (period start and first segment time), but in both snapshots, these appear to be the same segment. You should use the startNumber attribute to indicate the position of the first segment listed, and you should not change the period start time.

I recommend you change:

• Add startNumber and keep it updated to reflect the first segment listed in <SegmentTimeline>
• Make Period@start fixed (at 0, perhaps)
• Adjust your use of presentationTimeOffset with respect to your adjustment of Period@start
• Add suggestedPresentationDelay to cover the lag in serving the content

[hdodev]

Hi Joey,

First, thank you for your detailed analysis. This will allow us to move forward. OK, the "mixed-content" error is clear!

Just to be sure to understand correctly, your conclusion is that for a absolute moment, the identified segment is not the same between successive manifests, as our timestamps are changing? Correct me please if I'm wrong.

Then, I thought "startNumber" and "suggestedPresentationDelay" were DASH optional parameters, and that "Period@start" could not be fixed. So are they really required if we adjust the use of "presentationTimeOffset" with respect to our adjustment of "Period@start"? Correct me please if I'm wrong.

Thank you in advance (and sorry if my questions do not make much sense :-) ).

[joeyparrish]

startNumber is optional in some circumstances, but not in this case. If segments were never dropped from the list, startNumber could default to 1. But if your segment list changes, your startNumber must be updated to reflect that. Otherwise, there is no (easy) way to correlate one list to another to merge them.

suggestedPresentationDelay is truly optional, but highly recommended. You should decide what your presentation delay will be, to handle lag properly and keep clients from buffering at the edge. If you omit this from the manifest, we will provide a default. You can configure this default value at runtime with the key manifest.dash.defaultPresentationDelay.

Period@start must be fixed, so it is the other things which must be adjusted as time passes.

[joeyparrish]

Does this answer all your questions?

[hdodev]

Hi Joey, thank you for this new answer. It' helpfull yes :-).
I'll transfer this analysis and these additionnal notes to my Packager team, and check with them if they have more questions. If not it will be ok to close the item.

[joeyparrish]

Okay, sounds good. Just let us know when you've heard back.

[ismena]

@hdodev how did it go with the packager team? :) Is there anything else we can do for you on this issue?

[hdodev]

Hi @ismena,

The packager team agree with your analysis, and we made the modifications proposed.

We modified:

• Period@start fixed, but not 0 for the moment.
• PresentationTimeOffset is so calculated relatively to it.
• suggestedPresentationDelay is set 30 seconds.

But to be honest, it's not ok for the moment : When we make the player.load action, the playback doesn't start imediately :-(. We have to click on the Play button of the Player, and to move the cursor on the seek bar to make it start! However when started, it doesn't interrupt anymore :-).

Maybe you have an idea? As it's not the same symptom as previously, if you prefer us to close this issue we can. And if we don't solve it in our side, I could open another one to get your help. What do you prefer?

[hdodev]

Hi @shakateam,

Despite our attempts, we are still unable to playback the stream.
We tested with 2.3.0 version, 2.3.3 version, and the last 2.3.4 version.

We tried with 2.2.2 version too, and it works. Curiously, from 2.2.3 version we have the same comportment.

Can you please help us a last time?

Thanks in advance.

[joeyparrish]

We are working on a couple of issues that might be related. We have just discovered some corner cases in which playback doesn't start properly. You might be running into one of these same issues.

I would recommend trying again with our nightly build after we close #1298 and/or #1331.

Do you have an updated URL we can use to reproduce your issue with the updated streams? We could double-check for you if our fixes-in-progress for #1298 and #1331 will be effective for this issue, rather than you having to wait for the fixes for those to be released.

[joeyparrish]

#1298 and #1331 have been fixed. Can you try again with the latest code from master?

[hdodev]

Hi. It seems to work! I'm currently out of office, but I'll try to test with longer sessions, and make you a feedback when I'm back.

[joeyparrish]

Okay. We will mark as a duplicate and close the issue for now. If you find other failures, you can ask for the Shaka bot to reopen the issue by saying @shaka-bot reopen. The fixes for #1298 and #1331 will appear in v2.3.5. Thanks!

[hdodev]

OK, thanks for your help.

[hdodev]

@shaka-bot reopen

New tests done with 2.3.5 version.

• The playback start is now ok, and is not interrupted anymore during a certain duration.
• But, after about 40 minutes, the playback stops.

In the log, we can see there is is an infinite loop:
09:46:18.402 dash_parser.js:1186 Updating manifest...
09:46:18.425 streaming_engine.js:1133 (audio:6) timeNeeded=1522827918.1069446
09:46:18.426 streaming_engine.js:1144 (audio:6) update_: playheadTime=1522827917.649721 bufferedAhead=0.45722317695617676
09:46:18.427 streaming_engine.js:1295 (audio:6) next position known: position=2171
09:46:18.427 streaming_engine.js:1395 (audio:6) segment does not exist: currentPeriod.startTime=1522824320.072 position=2171
09:46:18.428 streaming_engine.js:2167 (audio:6) updating in 1 seconds
09:46:18.520 presentation_timeline.js:211 notifyNewSegmentDuration: maxSegmentDuration=2.2
09:46:18.520 dash_parser.js:1235 actual update period 3
09:46:19.314 streaming_engine.js:1133 (video:3) timeNeeded=1522827918.072
09:46:19.315 streaming_engine.js:1144 (video:3) update_: playheadTime=1522827917.649721 bufferedAhead=0.34227919578552246
09:46:19.316 streaming_engine.js:1295 (video:3) next position known: position=2171
...

For security reasons, the previous streams are now closed.
Here is a new one: http://195.36.159.232/testccast/bxu/manifest.mpd

[joeyparrish]

@hdodev, I can't access your stream due to CORS errors. Can you please add appropriate CORS headers in your web server?

[TheModMaker]

The CORS errors are caused by a redirect. If I just use the redirected URL directly, it works fine. The problem I see is that there is a gap between Periods, specifically there are not enough segments in one Representation. We support gaps between Periods, but only when the Period has an explicit duration. We may want to detect this and handle the gap automatically.

When I fetched the manifest, the first Period had an ID of 188269246292 and a start time of 1523290742.183. The segments for the video stream end at t="188528165492", which puts the presentation time at 1523293619.063. The second Period starts at 1523293652.680, which is 33.617 seconds away. So we can't find the segments from the end of the first Period.

[hdodev]

Hi @jacob, and thank you.
You confirm so you can reproduce it after about 40 minutes of playback?
So, what is the best thing to do according to you? Should we define an explicit period duration (does it make sense for Live streams?) or should we fixed the gap between periods? Or both?

[TheModMaker]

Hmm, I've been playing for almost 2 hours with no problem. Plus, I think we should actually handle the case of gaps between Periods already. I'll look into it more.

But you should avoid gaps between Periods if at all possible. Every Representation should have segments for the entire Period and each Period shouldn't have gaps between them.

[TheModMaker]

Now I no longer see a manifest with two Periods in it. When the Period transition happens, the new manifest only contains the second Period. This isn't wrong, but it breaks our gap-jumping logic since it only works if we see the two Periods in the same manifest. We may be able to add support for this case, but it may not be simple. Tentatively scheduled for v2.5.

But there are still gaps between the Periods. It would be ideal if you could fix them, but we can work to fix our gap-jumping logic.

[hdodev]

Hi @TheModMaker. I'm surprised by the fact that there were 2 periods in the same manifest with our current Packager configuration, and as it's a clear DASH stream. I suspect there was a problem during your first test, such as for example a temporary flow loss by packager.

Anyway, thanks again for your help, and for the enhancement proposal.

[davidloiret]

Any update?

[hdodev]

Hi,
As I didn't reproduce this problem yet even with a packager bascule, this case may be closed.
In a pinch, if it would happen again, I'll try with the suggested enhancement if effective in 2.5.