Codec Extensibility

[softworkz]

Problem

HLS.js lacks support for a number of codecs that are allowed by the HLS spec (RFC 6381 - not what Apple puts on top of it) and supported by certain browsers/platforms.

It has been declared in a number of cases (MPEG2 audio and video, AC-3 audio, HEVC video) that there are no intentions to include support for those codecs or that the use case would be unclear (HEVC).

Suggested Solution

Provide extension points in HLS.js that allow easy integration of additional codecs.
Maybe call it "codec plugins".

Alternatives you've considered

Maintaining such additions just as code patches is difficult to maintain and even harder to contribute publicly.

Additional context

We might be able to contribute in this area, updating some past efforts and adding some new.

[robwalch]

Hi @softworkz,

It has been declared in a number of cases (MPEG2 audio and video, AC-3 audio, HEVC video) that there are no intentions to include support for those codecs or that the use case would be unclear (HEVC).

That's not entirely true. Codec support is dependent on the browser and MSE, but in places where a browser may support HEVC or AC-3 via MSE, hls.js's demuxers/remuxers may not at this point.

The best place to provide input is in the individual tickets that we are tracking in the Release Planning and Backlog project. They include:

• HEVC support HEVC support #2451 (issue)
• AC-3 support add support for ac-3 codec #1102 (PR)
• And a placeholder for describing how multi-codec main playlist should be handled (both when MSE can and cannot handle codec switching midstream).

There are no plans for a plugin system, and no proof that creating one would make the work any easier.

[softworkz]

There are no plans for a plugin system, and no proof that creating one would make the work any easier.

In the AC-3 conversation there were concerns about the size that this would add to the distribution and plugins/modules were mentioned there as well. That's why I thought that this might be the reason why there hasn't been any progress.

• And a placeholder for describing how multi-codec main playlist should be handled (both when MSE can and cannot handle codec switching midstream).

Ah great - I hadn't seen that. That's something we'd be interested as well.

[robwalch]

In the AC-3 conversation there were concerns about the size that this would add to the distribution and plugins/modules were mentioned there as well. That's why I thought that this might be the reason why there hasn't been any progress.

Good point and noted. I think that was more of a concern back in 2017 that blocked that PR specifically. It doesn't look like it would have that dramatic of an impact on library size. The larger concerns I have are supporting it in fmp4, support checks and preventing codec switches that could cause decode errors.

[softworkz]

We've integrated this a while ago and afaik, there haven't been any issues, but I'll check back.