Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Account for extended MIME type parameters in MediaCapabilities.decodingInfo() polyfill for Cast platform #4726

Closed
JulianDomingo opened this issue Nov 17, 2022 · 0 comments · Fixed by #4727
Assignees
Labels
platform: Cast Issues affecting Cast devices priority: P1 Big impact or workaround impractical; resolve before feature release status: archived Archived and locked; will not be updated type: bug Something isn't working correctly
Milestone

Comments

@JulianDomingo
Copy link
Contributor

JulianDomingo commented Nov 17, 2022

Have you read the FAQ and checked for duplicate open issues?
Yes.

What version of Shaka Player are you using?
Latest (v4.3.0 as of writing).

Can you reproduce the issue with our latest release version?
Yes.

Can you reproduce the issue with the latest code from main?
Yes.

Are you using the demo app or your own custom app?
Custom app (Cast Receiver)

If custom app, can you reproduce the issue using our demo app?
Yes.

What browser and OS are you using?
Chrome, Cast OS

For embedded devices (smart TVs, etc.), what model and firmware version are you using?
Google Nest Hub / FW Version: 1.56.271947

What are the manifest and license server URIs?

https://cph-p2p-msl.akamaized.net/hls/live/2000341/test/master.m3u8

What configuration are you using? What is the output of player.getConfiguration()?

Default Cast player configuration set in the receiver SDK. Since these are privately set, refraining from posting them.

What did you do?

  1. Cast above live HLS stream on a Google Nest Hub. Cast debugging typically involves use of https://casttool.appspot.com/cactool/ when testing non-Cast partner and/or publicly available streams.

What did you expect to happen?

  1. Cast above live HLS stream on a Google Nest Hub. The live stream has unsupported stream variants (too large of a resolution).
  2. Google Nest Hub (with max resolution 1280x720) will select and play only supported stream variant(s) (e.g., anything with a resolution <= 1280x720).

What actually happened?

  1. Cast above live HLS stream on a Google Nest Hub. The live stream has unsupported stream variants (too large of a resolution).
  2. Google Nest Hub (with max resolution 1280x720) will select and play an unsupported stream variant (e.g., a 4K stream). This hinders video startup latency.

Shaka’s polyfill implementation of MediaCapabilities.decondingInfo() as of writing does NOT leverage the variant’s resolution, frame rate, or eotf support to make decisions on whether or not a stream is supported. E.g., streams which report larger resolutions than what the Cast platform reports it can handle would incorrectly bypass Shaka’s filtering by MediaCapabilities if all other stream parameters are supported, like MIME type.

Within the decodingInfo() polyfill, Shaka checks for variant support first through MediaSource.isTypeSupported(contentType):

const isSupported = MediaSource.isTypeSupported(contentType);

Within the context of decodingInfo, contentType consists only of a MIME type and codecs parameter since this is what the API instructs: https://developer.mozilla.org/en-US/docs/Web/API/MediaCapabilities/decodingInfo.

The Cast polyfill for MediaSource.isTypeSupported() doesn't manipulate contentType either:

// Only canDisplayType can check extended MIME type parameters on this
// platform (such as frame rate, resolution, etc).
// In previous versions of this polyfill, the MIME type parameters were
// manipulated, filtered, or extended. This is no longer true, so we pass
// the full MIME type to the platform as we received it.
return cast.__platform__.canDisplayType(mimeType);
};

Therefore, Shaka should add a conditional check within the MediaCapabilities.decodingInfo() polyfill for Cast platforms to instead rely on cast.__platform__.canDisplayType() which does accept extended MIME type parameters to determine proper stream variant support.

@JulianDomingo JulianDomingo added the type: bug Something isn't working correctly label Nov 17, 2022
@JulianDomingo JulianDomingo self-assigned this Nov 17, 2022
@JulianDomingo JulianDomingo added this to the v4.4 milestone Nov 17, 2022
@avelad avelad added priority: P1 Big impact or workaround impractical; resolve before feature release platform: Cast Issues affecting Cast devices labels Nov 17, 2022
joeyparrish pushed a commit that referenced this issue Dec 1, 2022
See #4726 for more
context.

This allows Cast devices to properly filter stream variants with a
resolution surpassing that of the device's capabilities.

We place the fix in the `MediaCapabilities` polyfill since it's intended
to be the right way to check for anything related to platform support.

HDR support checks will require `eotf=smpte2048`, as indicated in
#2813 (comment).
Specifically, a `{hev|hvc}1.2` profile is only an *indication* of an HDR
transfer function, but *may* be a non-HDR 10-bit color stream.

In Cast, the platform can distinguish between the two by explicitly
providing the transfer function; it uses `smpte2048` (`"PQ"`) because
this is the "basis of HDR video formats..."
(https://en.wikipedia.org/wiki/Perceptual_quantizer).
joeyparrish pushed a commit that referenced this issue Dec 8, 2022
See #4726 for more
context.

This allows Cast devices to properly filter stream variants with a
resolution surpassing that of the device's capabilities.

We place the fix in the `MediaCapabilities` polyfill since it's intended
to be the right way to check for anything related to platform support.

HDR support checks will require `eotf=smpte2048`, as indicated in
#2813 (comment).
Specifically, a `{hev|hvc}1.2` profile is only an *indication* of an HDR
transfer function, but *may* be a non-HDR 10-bit color stream.

In Cast, the platform can distinguish between the two by explicitly
providing the transfer function; it uses `smpte2048` (`"PQ"`) because
this is the "basis of HDR video formats..."
(https://en.wikipedia.org/wiki/Perceptual_quantizer).
joeyparrish pushed a commit that referenced this issue Dec 8, 2022
See #4726 for more
context.

This allows Cast devices to properly filter stream variants with a
resolution surpassing that of the device's capabilities.

We place the fix in the `MediaCapabilities` polyfill since it's intended
to be the right way to check for anything related to platform support.

HDR support checks will require `eotf=smpte2048`, as indicated in
#2813 (comment).
Specifically, a `{hev|hvc}1.2` profile is only an *indication* of an HDR
transfer function, but *may* be a non-HDR 10-bit color stream.

In Cast, the platform can distinguish between the two by explicitly
providing the transfer function; it uses `smpte2048` (`"PQ"`) because
this is the "basis of HDR video formats..."
(https://en.wikipedia.org/wiki/Perceptual_quantizer).
joeyparrish pushed a commit that referenced this issue Dec 8, 2022
See #4726 for more
context.

This allows Cast devices to properly filter stream variants with a
resolution surpassing that of the device's capabilities.

We place the fix in the `MediaCapabilities` polyfill since it's intended
to be the right way to check for anything related to platform support.

HDR support checks will require `eotf=smpte2048`, as indicated in
#2813 (comment).
Specifically, a `{hev|hvc}1.2` profile is only an *indication* of an HDR
transfer function, but *may* be a non-HDR 10-bit color stream.

In Cast, the platform can distinguish between the two by explicitly
providing the transfer function; it uses `smpte2048` (`"PQ"`) because
this is the "basis of HDR video formats..."
(https://en.wikipedia.org/wiki/Perceptual_quantizer).
joeyparrish pushed a commit that referenced this issue Dec 8, 2022
See #4726 for more
context.

This allows Cast devices to properly filter stream variants with a
resolution surpassing that of the device's capabilities.

We place the fix in the `MediaCapabilities` polyfill since it's intended
to be the right way to check for anything related to platform support.

HDR support checks will require `eotf=smpte2048`, as indicated in
#2813 (comment).
Specifically, a `{hev|hvc}1.2` profile is only an *indication* of an HDR
transfer function, but *may* be a non-HDR 10-bit color stream.

In Cast, the platform can distinguish between the two by explicitly
providing the transfer function; it uses `smpte2048` (`"PQ"`) because
this is the "basis of HDR video formats..."
(https://en.wikipedia.org/wiki/Perceptual_quantizer).
@github-actions github-actions bot added the status: archived Archived and locked; will not be updated label Jan 30, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 30, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
platform: Cast Issues affecting Cast devices priority: P1 Big impact or workaround impractical; resolve before feature release status: archived Archived and locked; will not be updated type: bug Something isn't working correctly
Projects
None yet
2 participants