Allow createEncodedStreams on PCs without encodedInsertableStreams parameter

[kjvenalainen]

Feature Request

Background

Mediasoup supports an additionalSetting on transports encodedInsertableStreams: true which is a Chromium-specific extension allowing for an equivalent to RTCRTPScriptTransform to be added to senders and receivers.

When encodedInsertableStreams is true, applications can call createEncodedStreams() on RTCRtpReceiver/RTCRtpSender at any time after creation. Until this function is called, all packets in the associated streams are dropped. Thus, in order to flow data, a RTCRTPScriptTransform must be registered for each stream coming from the peerconnection. This is unfortunate, because there is performance overhead from these streams and there is no way to not-set a transform (for example, if we don't want to transform video streams).

Here the W3 spec: https://www.w3.org/TR/webrtc-encoded-transform/

What's New?

Starting with Chrome 121 (~Feb 2024), the Google team landed a change Allow createEncodedStreams on PCs without encodedInsertableStreams param which, as the title says, allows creating a RTCRTPScriptTransform on streams created from PCs that don't set the encodedInsertableStreams flag. This is perfect, since we can register transforms only for the actual streams which need it.

The limitation that comes along with the new capability is that if a transform is needed, createInsertableStreams() must be called synchronously "within one JS event loop spin [of creating the receiver/sender]".

Problem

mediasoup-client abstracts the process of sender/receiver creation behind async Transport.producer/receive() functions. Thus, by the time these functions return it is too late to register the transform. Indeed Chrome gives the following error: InvalidStateError: Failed to execute 'createEncodedStreams' on 'RTCRtpReceiver': Too late to create encoded streams.

The proper place where createEncodedStreams() must be called (for a receiver) is here:

mediasoup-client/src/handlers/Chrome111.ts

Line 787 in 470c90d

await this._pc.setRemoteDescription(offer);

, just after the line:

// src/handlers/Chrome111.ts @ receive()

await this._pc.setRemoteDescription(offer);

// NEW: Create the transform streams.
for (const options of optionsList) {
    const { trackId } = options;
    const localId = mapLocalId.get(trackId);
    const transceiver = this._pc.getTransceivers()
        .find((t) => t.mid === localId);
    if (!transceiver) {
        throw new Error('transceiver not found');
    }
    const {
        readable,
        writable,
    } = transceiver.receiver.createEncodedStreams();
}

But there is the problem of how to hand the readable, writable streams back out to the caller. Additionally, this is a Chromium-specific API and therefore these streams will never exist on Firefox nor Safari.

One alternative is to hook from the application into the track event of the internal PC before calling receive(), but this goes against existing mediasoup API design:

// Application code, during transport creation.

newTransport.handler._pc.addEventListener(
  'track',
  (event) => {
    const {
      readable,
      writable,
    } = event.transceiver.receiver.createEncodedStreams();
  },
);

Proposed Solution

We could add a new callback to ConsumerOptions so the application can pass an optional onRtpReceiver callback:

const consumer = await recvTransport.consume({
  xxxxx,
  xxxxx,
  onRtpReceiver: (receiver) => {
    const { readable, writable } = receiver.createEncodedStreams();
    // do things with them
  }
});

In Chrome handler:

await this._pc.setRemoteDescription(offer);

for (const options of optionsList) {
  const { trackId, onRtpReceiver } = options;

  if (onRtpReceiver) {
    const localId = mapLocalId.get(trackId);
    const transceiver = this._pc.getTransceivers()
      .find((t) => t.mid === localId);
    
    if (!transceiver) {
      throw new Error('transceiver not found');
    }

    onRtpReceiver(transceiver.receiver);
  }
}

Also, add similar logic to the sendTransport.produce function for registering sender side equivalents.

[ibc]

Implemented in mediasoup-client 3.7.10:

• https://mediasoup.org/documentation/v3/mediasoup-client/api/#ProducerOptions
• https://mediasoup.org/documentation/v3/mediasoup-client/api/#ConsumerOptions