DASH-HLS Interop: Image-based subtitles and trickplay tracks

[nicoweilelemental]

Image-based subtitles tracks
For workflow reasons and charset reasons, some content owners don't include text-based subtitles in the live channels sources that they provide to distributors, but rather image-based subtitles (like DVB-Sub). While it's possible to transform the subtitles as IMSC1 Image Profile as per DASH-IF IOP section 6.4.4, there is no equivalent IMSC1 Image Profile support in the HLS RFC, which means that companies will continue to rely on proprietary forks of the RFC to support these use cases. Could we discuss the integration in the HLS RFC with Apple? Even if it wasn't supported by Apple players, it would be tremendously helpful for interoperability in the rest of the video ecosystem.

Image-based trickplay tracks
For player resources optimization reasons, the use of a video track as a trickplay artefact is not always possible, and a lot of player providers recommend the use of image thumbnails tracks instead of special low framerate video tracks. DASHIF IOP section 6.2.6 covers this use case but there is equivalent support in the HLS RFC. There is the Image media playlists HLS extension proposal from Roku/Disney/Warner here https://developer.roku.com/docs/developer-program/media-playback/trick-mode/hls-and-dash.md but its relevance/adoption is limited by the fact that it's not part of the RFC. Same logic here: even if not supported by Apple players which don't need it as they can leverage iframe-only tracks, it would be super useful for the rest of the ecosystem to get this officially part of the HLS RFC.

[nicoweilelemental]

As an alternative to the direct integration into the HLS RFC, we could imagine to see CTA-WAVE authoring and maintaining an official HLS-Extensions RFC, for non-Apple payers. This would already be a huge improvement compared to the current situation, with multiple HLS forks and extensions proposals scattered everywhere.

[mcpv]

Here's the spec we wrote with Roku. We could register it with IETF as an HLS extension so it could be referenced for Interop specs.

https://github.com/image-media-playlist/spec

[billmay]

Instead of that, I'd rather take it to the new hls-interest group with the ietf, see if Apple will make it more official.

[nicoweilelemental]

It's obviously better if new features make their way in the RFC. If Apple doesn't agree to introduce it, then we still can discuss with them a coordinated approach where an HLS-Extensions RFC (not only with image-based subtitles and media playlists, but with a wider scope) lives alongside the HLS core RFC. I will send the request to the mailing list.

[krasimirkolarov]

Hi Nicolas, Please do. That is exactly the purpose of the IETF list. Best, Krasimir

…

On May 20, 2020, at 10:28 AM, Nicolas Weil ***@***.***> wrote: It's obviously better if new features make their way in the RFC. If Apple doesn't agree to introduce it, then we still can discuss with them a coordinated approach where an HLS-Extensions RFC (not only with image-based subtitles and media playlists, but with a wider scope) lives alongside the HLS core RFC. I will send the request to the mailing list. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/cta-wave/content-specification-task-force/issues/47#issuecomment-631616623>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEM2GTTC3TYVCLDJIUTGJ7TRSQHLLANCNFSM4M4MAOHQ>.

[technogeek00]

Previous IETF thread on image-based subtitles and trickplay tracks: https://mailarchive.ietf.org/arch/msg/hls-interest/FCfPmXIU6YApjrH7UltpDVSezTA/

[technogeek00]

@technogeek00 to split into two separate issues to better track the differing tracks of work.

[nicoweilelemental]

We now have reference live streams with Image-based trickplay tracks, for both HLS and DASH, I will add it to the new issue, once created.

[technogeek00]

Image based subtitle tracks

Question	Answer
Does the feature relate to an industry streaming use-case?	Yes, Subtitles
- What is the commonality of this case?	Uncommon
- Is this an established or emerging practice?	Declining, movement towards font-based
Does this feature have related mechanisms in both DASH and HLS?	No
- What is the maturity of support in both specifications?	DASH - Mature, HLS - Missing
- What is the maturity of implementation support for both specifications?	DASH - Immature, HLS - N/A
- Are there known interoperability issues between specs?	Yes, HLS explicitly excludes
- Has anyone implemented this interoperably?	Potentially
- Are there features missing in a specification with open proposals for it?	Potentially
Has the industry defined defacto mechanisms not present in both DASH and/or HLS?	Yes
- Why was functionality defined outside of the main specifications?	Usage of image-based IMSC in HLS
- Has the functionality been standardized elsewhere (DASH-IF, CTA, SVA)?	No
- Is the functionality proprietary or openly developed?	Proprietary
- Could the functionality be incorporated into specifications with evangelism?	No, clear stance from Apple

Image-based trickplay tracks

Question	Answer
Does the feature relate to an industry streaming use-case?	Yes, Thumbnails / Trickplay
- What is the commonality of this case?	Very Common
- Is this an established or emerging practice?	Established
Does this feature have related mechanisms in both DASH and HLS?	No
- What is the maturity of support in both specifications?	DASH - Mature, HLS - Missing
- What is the maturity of implementation support for both specifications?	DASH - Mature, HLS - N/A
- Are there known interoperability issues between specs?	Yes, HLS does not support
- Has anyone implemented this interoperably?	Yes
- Are there features missing in a specification with open proposals for it?	Yes
Has the industry defined defacto mechanisms not present in both DASH and/or HLS?	Yes
- Why was functionality defined outside of the main specifications?	Industry desire
- Has the functionality been standardized elsewhere (DASH-IF, CTA, SVA)?	No
- Is the functionality proprietary or openly developed?	Proprietary, but open
- Could the functionality be incorporated into specifications with evangelism?	Yes

[nicoweilelemental]

Image-based trickplay tracks
HLS image-based trickplay track was already a fairly common feature for VOD contents targeting the Roku platform. Now it's spreading in the live domain too, as the spec extension is simple and efficient, and there is a growing demand for it.

Here is an AWS Elemental stream showing the implementation of the Image Media Playlist spec for the live use case: https://d345o62q7f2dwl.cloudfront.net/referencestream/main.m3u8

It's fairly easy to configure with MediaLive + MediaStore, or MediaLive + S3: https://docs.aws.amazon.com/medialive/latest/ug/trick-play-roku.html

The implementation work is happening on MediaPackage too, as we speak, and in the player ecosystem. That would be really great to capitalize on the Roku/Disney/Warner spec, which is the de factor industry standard now.

Image-based subtitle tracks
I agree with the assessment that this is a declining need, mostly related to legacy playback environments. We recently introduced DVB-Sub passthrough for HLS/TS on MediaPackage, but right now we don't see customer demand for forward looking formats like IMSC Image Profile. AS an industry, we should probably not spend more time on it until there is a precise need identified.

[technogeek00]

Issues split

• Image-based trickplay tracks #37 for Trickplay tracks
• Image-based subtitle tracks #38 for subtitle tracks