WebRTC peer connections and codecs

[jgraham]

Description

Video conferencing requires interoperability beyond compatible runtime environments, as it needs to solve negotiation and establishment of runtime audiovisual communication with other user agents directly. Sites and platforms have different codec needs, requiring APIs for customization.

Specification

https://w3c.github.io/webrtc-pc/

Open Issues

No response

Tests

• Anything with “codec” in https://wpt.fyi/results/webrtc
• Anything with “codec” in https://wpt.fyi/results/webrtc-extensions
• Anything with “codec” in https://wpt.fyi/results/webrtc/simulcast

Current Implementations

• Blink
• Gecko
• WebKit

Standards Positions

No response

Browser bug reports

No response

Developer discussions

No response

Polls & Surveys

No response

Existing Usage

No response

Workarounds

No response

Accessibility Impact

No response

Privacy Impact

No response

Other

Note that WebRTC in general does not show up as a priority on developer surveys as it's generally used by a small number of sites that specialize in video conferencing and related services. However those sites are often very popular and compatibility issues highly visible to users (see e.g. webcompat.com issue reports)

[jgraham]

CC @jan-ivar

[foolip]

For survey data and web developer demand, in preliminary results from State of HTML 2023, WebRTC was a somewhat common response to the freeform question "Which existing HTML features or browser APIs are you unable to use because of browser differences or lack of support?"

[alvestrand]

The common requirement in WebRTC is that browsers MUST support H.264 Constrained Baseline and VP8.
The space of possible H.264 profiles is quite confusing, and there are parameters (max-fs and max-fr) that are usually not generated or parsed for VP8.

Things that could be worked on:

• Verifying that the mandatory profiles are indeed supported by all browsers
• Testing the generation and effect of mandatory-to-implement parameters
• Surveying the implementation of other codecs and profiles, to see if there's anything that can reasonably be added to the baseline requirements
• Testing the behavior of codecs under stress, such as when experiencing limited bandwidth or packet loss

There's a lot of other behaviors that could be tested, but are not so directly related to codecs.

[alvestrand]

Note that web-platform-tests/wpt#36487 - a proposal that allows observing bits on the wire - would make testing codec bitstreams easier.

[foolip]

@jgraham can you confirm if the existing tests are enough to be useful here, or if testing of specific codec bitstreams using a mocked peer is necessary?

[fippo]

Since codecs beyond the MTI VP8 and H264 are optional... can we add tests for codec support gated by assert_precondition?
Essentially extensions to https://wpt.live/webrtc/protocol/video-codecs.https.html

[gsnedders]

Since codecs beyond the MTI VP8 and H264 are optional... can we add tests for codec support gated by assert_precondition?

#375 had a long debate about whether we want tests with assert_precondition in Interop.

[jensimmons]

Tests
Anything with “codec” in https://wpt.fyi/results/webrtc
Anything with “codec” in https://wpt.fyi/results/webrtc-extensions
Anything with “codec” in https://wpt.fyi/results/webrtc/simulcast

Do you mean this?

• https://wpt.fyi/results/webrtc?label=master&label=experimental&aligned&q=codec
• https://wpt.fyi/results/webrtc-extensions?label=master&label=experimental&aligned&q=codec
• https://wpt.fyi/results/webrtc/simulcast?label=master&label=experimental&aligned&q=codec (but there's nothing here)

Any way to create a better link to the specific tests you are suggesting?

[jan-ivar]

Do you mean this?

• https://wpt.fyi/results/webrtc?label=master&label=experimental&aligned&q=codec
• https://wpt.fyi/results/webrtc-extensions?label=master&label=experimental&aligned&q=codec

Yes, thanks for the direct links!

As to scope, I don't think we meant to preclude "codec" tests yet to be written or improved, if there's interest — in practice we find our tests grow as more implementors implement — AFAIK our WebRTC team(s) haven't participated in interop-20xx before, so we're open to input on how to best organize this.

Anything with “codec” in https://wpt.fyi/results/webrtc/simulcast ... (but there's nothing here)

Sorry, that should have read anything with "h264", "vp8" or "vp9" (we can leave out "vp9" if need be). I originally had "anything codec-related" in any of these specs, but the format of the nomination process encouraged being specific.

[jan-ivar]

There's a lot of other behaviors that could be tested, but are not so directly related to codecs.

I'm open to broadening the scope if there's interest. Just happy to have WebRTC in the running. 🙂

[gsnedders]

Any way to create a better link to the specific tests you are suggesting?

https://wpt.fyi/results/?label=master&label=experimental&aligned&q=%28path%3A%2Fwebrtc%20or%20path%3A%2Fwebrtc-extensions%29%20codec

[nairnandu]

Thank you for proposing WebRTC peer connections and codecs for inclusion in Interop 2024.

We wanted to let you know that this proposal was not selected to be part of Interop 2024. This is because we got many more proposals than we could include in this year's project. Note that individual vendors may nevertheless choose to advance work in this area during the forthcoming year. We would welcome this proposal being resubmitted again next year, if necessary.

For an overview of our process, see proposal selection. Thank you again for contributing to Interop 2024!

Posted on behalf of the Interop team.