Support smooth stream manifest parsing

[OrenMe]

Are there any plans to support parsing and playing smooth stream over MSE?
It is fmp4 container and it is possible to create CENC pssh from the playready protection header data.
Also parsing dfxp to webvtt is possible with adequate support for style.

Would appreciate any pointers if this is even possible and how to get started with it.
Thanks for all this great work!

[ismena]

@OrenMe thanks for reaching out.
Let me make sure I understand the question: are we talking Microsoft Smooth Streaming?

[OrenMe]

Hi @ismena, yes.

[ismena]

Got you! We have recently started adding support for HLS and it's possible we will proceed with implementing support for smooth streaming as well in the future, but it's not on our road map at the moment.

I haven't closely looked at the format, so please take this as an educated guess: our approach to HLS has been to write a parser plugin that parses the manifest and hooks it up with already existing Shaka streaming logic. I think that might the way to go with smooth streaming as well.
You can take a look at how dash and hls parsers do they work and integrate with streaming in lib/dash and lib/hls if that helps.

[OrenMe]

Thanks @ismena, would definitely check it out and see if I can contribute back.

[ToreJuloe]

Dash.js seems to be working on this as well. Perhaps there's some insight to be gained from reading their parser: https://github.com/Dash-Industry-Forum/dash.js/blob/development/src/mss/parser/MssParser.js

rx-player also has one: https://github.com/canalplus/rx-player/blob/master/src/net/smooth/parser.js

[ismena]

@ToreJuloe thanks for the links! We'll be sure to take a look if we start working on this. It also could be useful if someone wants to do a PR on this.

[zangue]

I started working on this lately and I would be happy to submit a PR once ready.

[pmcxs]

@zangue, what's the status of that PR?

[olivas-bo]

I started working on this as well. I have a question regarding the MSS fragments. AFAIK, the segments are not compatible with MSE, also, MSS does not have the concept of the init segment, so i will have to create a virtual init segment (similar to what hasplayer is doing) and transform the fetched segments. In order to do this is there a clean way to intercept the parsing of the segment before appending to the SourceBuffer?

Thanks in advance.

[joeyparrish]

In the manifest structure, each Stream has an InitSegmentReference. If the segments are self-initializing, you can leave it null. Otherwise, you will need to construct a valid reference to a valid init segment.

If you can construct a valid MP4 init segment in the manifest parser, you could set the URI in the InitSegmentReference to a data URI containing the init segment.

If you would rather not spend the memory upfront and you'd rather construct them on-the-fly, you could always do something a little more complicated. You could register a scheme handler with NetworkingEngine, construct a URI of your own devising for each init segment which points back to your scheme, and then your scheme handler plugin can construct the init segment when requested.

I strongly advise the data URI approach, unless you run into serious issues with it.

[olivas-bo]

Thanks @joeyparrish i will follow your advise and try the data URI approach.

[zangue]

@pmcxs Sorry for the late reply. Essentially, manifest parsing is done, Init segment generation is almost done. What is left is patching MSS media segments to conform with the MSE, DRM support and writing tests (so, quite a bit actually). Work is currently on hold cause I have been lacking spare time lately but it looks like @olivas-bo is taking the lead on this issue anyway :)

Btw @olivas-bo to address the second point of your question about a clean way to intercept and patch media segments before they are appended to the source buffer, I would suggest you use a ReponseFilter.

[olivas-bo]

Thanks @zangue this was exactly what i was looking for.

[joeyparrish]

JFYI, we are planning some changes to the internal manifest structure in #1339. This would impact your work on SmoothStreaming.

I'll let you know once we start work on that, so you can keep pace with the changes we are making. Mostly, you will just need to move some properties to other objects.

Does SmoothStreaming have an equivalent to DASH's periods? Or HLS's discontinuities?

[subhranshudas]

I am actually new to the Streaming world, not sure if this is the right place to ask the question. I am looking to support MSS in shaka-player...from what i have read so far, this thread is trying to resolve that issue. May i know the ETA for MSS support on Shaka Player (how soon can we expect...). Also is there an official industry standard documentation to write the implementation for any DASH player for the MSS format. And also do we have any comparative analysis as to why shaka should be preferred over dash.js or vice versa. Thank you!!

[joeyparrish]

DASH and MSS are two independent streaming formats, so a DASH player isn't necessarily also an MSS player. If you would like start a discussion comparing various players and their features, please file a new issue. This one is about implementing MSS in Shaka Player.

As for ETA, the contributors working on that feature will have to answer. The Shaka Player team are not working on this one ourselves right now.

[subhranshudas]

@joeyparrish thank you for the reply. I actually meant "Adaptive client player". Thank you for clarifying shaka player team's current stance on MSS. I will definitely open a new issue for the comparison.

@olivas-bo @zangue guys i am looking to start to implement MSS support for shaka player, can you please let me know a good place to start (from specs wise, i am new to the streaming world, still exploring). Would appreciate some sources.

[joeyparrish]

I think it would be best for one person or group of people to complete a working MSS implementation and then send a pull request. @subhranshudas, it would not make sense for you to start working on an independent MSS implementation of your own.

[subhranshudas]

@joeyparrish i understand that the best way forward is to wait for the working implementation of MSS. But am not sure how long the wait can go on...so i guess i will have to start some where for my project requirements. Probably start dash.js & hasPlayer as sources.
If other contributors can share their thoughts & progress it would help me in making the choice.

[olivas-bo]

@joeyparrish sorry i have been away for some days. Thank you for your info, i will keep an eye on #1339.

Regarding your question about SmoothStreaming having an equivalent DASH's periods or HLS's discontinuities, the closest thing is called sparse tracks and it is used mainly to signal ads insertion. Does this answer your question?

@subhranshudas i will not commit on a date for this sorry...

[subhranshudas]

@olivas-bo thank you for the reply...will keep an eye for sure...

[jlnabais]

Hello all,

I've also been working in this along with @olivas-bo and we already got mss manifest, init segment generation and fragments apparently working (at least for the tested assets), at this point we can successfully playback widevine protected audio and video

However, for some reason, we are having a Shaka Error MANIFEST.RESTRICTIONS_CANNOT_BE_MET () (does not interrupts playback) error thrown, seems that for some reason the variants are getting empty internally and the code throws at the player module when checking if (!variants || !variants.length), we've noticed that this code gets executed twice and at a first run the variants are correctly filled, but empty in a second run, note that in the mss parser returned object they are correct. Following the call stack we get to processKeyStatusChanges_. However, so far, we are unable to understand the reason for variants to be empty at a certain point in time. @joeyparrish can you shed some light into this?

Many Thanks.

[chrisfillmore]

@jlnabais What browser are you testing in?

[jlnabais]

@chrisfillmore Chrome Version 65.0.3325.181 (Official Build) (64-bit)

[jlnabais]

Got it, variant.allowedByKeySystem was being defined as false, because we were wrongly setting KeyId when parsing from MSS manifest ProtectionHeader element. That would make the keyStatus to be false when running:

variant.allowedByKeySystem = !!keyStatus && restrictedStatuses.indexOf(keyStatus) < 0;

Therefore the variants would be empty. :)

[rgainza]

From the future. Has this been finished? @jlnabais

[joeyparrish]

From the future. Has this been finished? @jlnabais

What year are you from? :-)

No, this issue is still open, and Microsoft Smooth Streaming is not supported in Shaka Player. We haven't seen a PR to implement it, and it's not a priority feature for the core Shaka Player team.