Cannot detect when a track is added by the remote peer #1135

evan-hilscher · 2024-06-27T01:12:56Z

Similar to #983.

I am attempting to add a media track some time after the peer connection has been established (this code is executed by the local peer):

var localVideoTrack = new MediaStreamTrack(AudioVideoWellKnown.WellKnownVideoFormats[SDPWellKnownMediaFormatsEnum.JPEG], MediaStreamStatusEnum.SendOnly);
_peerConnection.AddTrack(localVideoTrack);

Renegotiation occurs as expected, but at no point is the remote peer informed that the local peer has added a track. I have subscribed to and am logging every RTCPeerConnection event (including OnRtpPacketReceivedByIndex as mentioned in #983), but none of them are raised on the remote peer.

Attempting to send any data over the track results in this logged warning, despite creating the track as SendOnly:

20:08:40 warn: sipsorcery[0] SendRtpRaw was called for an video packet on an RTP session with a Stream Status set to Inactive

The text was updated successfully, but these errors were encountered:

ChristopheI · 2024-09-04T08:55:01Z

Both peers are using SIPSorcery ?
Are you sure SDP are well exchanged between peers ?
Did you check SDP offers and answers content ?

evan-hilscher · 2024-09-05T22:25:40Z

Both peers are using SIPSorcery. I have verified that the offer SDP contains a new video track.

The issue appears to be here:

sipsorcery/src/net/RTP/RTPSession.cs

Lines 1015 to 1021 in 06f8e3c

    
           List<SDPAudioVideoMediaFormat> capabilities = null; 
        
           if (currentMediaStream.LocalTrack == null) 
        
           { 
        
               capabilities = remoteTrack.Capabilities; 
        
               var inactiveLocalTrack = new MediaStreamTrack(currentMediaStream.MediaType, false, remoteTrack.Capabilities, MediaStreamStatusEnum.Inactive); 
        
               currentMediaStream.LocalTrack = inactiveLocalTrack; 
        
           }

When the remote peer sends a media track that the local peer does not know about, it creates a new MediaStream and sets the LocalTrack of that stream to Inactive. I am not a WebRTC expert, but I think that perhaps an ontrack event should be raised somewhere near the referenced code, and that the local track should not be created as inactive.

ChristopheI · 2024-09-06T06:45:34Z

Could you provide all SDP exchanged from both sides ? (and from the beginning including the creation of the PeerConnection)

PeerA is the one adding the video track: So the SDP (as offer) sent must include two media (the one used to create the PeerConnection) and the new video with SendRecv or SendOnly

Once this SDP is received on PeerB, it must add a Video MediaTrack with RecvOnly or SendRecv according your needs and create an SDP (as anwser) and send it to PeerA

evan-hilscher · 2024-09-09T20:12:56Z

Here is the sequence of SDP exchanges:

Local peer receiving offer:

v=0
o=- 43754 0 IN IP4 127.0.0.1
s=sipsorcery
t=0 0
a=group:BUNDLE 0
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:XSOI
a=ice-pwd:ETVICNGSOOBXHCHIXVYUUGDH
a=fingerprint:sha-256 A2:2C:21:02:5C:9E:A7:C7:47:E1:C9:ED:13:C5:6F:6E:96:71:21:C1:69:DB:9C:69:60:68:20:4A:4E:92:F2:BA
a=setup:actpass
a=candidate:3913191201 1 udp 8447 157.245.114.91 58553 typ relay raddr 0.0.0.0 rport 0 generation 0
a=candidate:2564284589 1 udp 1677730047 75.100.125.142 45834 typ srflx raddr 0.0.0.0 rport 0 generation 0
a=candidate:2422434839 1 udp 2113937663 10.0.3.122 63054 typ host generation 0
a=ice-options:ice2,trickle
a=mid:0
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=end-of-candidates
a=sctp-port:5000
a=max-message-size:262144

Remote peer receiving answer:

v=0
o=- 32058 0 IN IP4 127.0.0.1
s=sipsorcery
t=0 0
a=group:BUNDLE 0
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:JRJJ
a=ice-pwd:DNVTQRUDPANEWTNZBVOJFZOC
a=fingerprint:sha-256 5D:1D:39:7C:FE:29:91:4E:C8:6C:CF:5B:55:D1:D2:BC:38:5E:3C:AF:79:41:4D:82:32:F0:6E:A4:E7:A1:DD:C6
a=setup:active
a=candidate:3913191201 1 udp 8447 157.245.114.91 59754 typ relay raddr 0.0.0.0 rport 0 generation 0
a=candidate:2564284589 1 udp 1677730047 75.100.125.142 52771 typ srflx raddr 0.0.0.0 rport 0 generation 0
a=candidate:2422434839 1 udp 2113937663 10.0.3.122 61986 typ host generation 0
a=ice-options:ice2,trickle
a=mid:0
a=end-of-candidates
a=sctp-port:5000
a=max-message-size:262144

Local peer receiving offer, with new video track:

v=0
o=- 43754 0 IN IP4 127.0.0.1
s=sipsorcery
t=0 0
a=group:BUNDLE 0 1
m=video 9 UDP/TLS/RTP/SAVP 26
c=IN IP4 0.0.0.0
a=ice-ufrag:XSOI
a=ice-pwd:ETVICNGSOOBXHCHIXVYUUGDH
a=fingerprint:sha-256 A2:2C:21:02:5C:9E:A7:C7:47:E1:C9:ED:13:C5:6F:6E:96:71:21:C1:69:DB:9C:69:60:68:20:4A:4E:92:F2:BA
a=setup:passive
a=candidate:3913191201 1 udp 8447 157.245.114.91 58553 typ relay raddr 0.0.0.0 rport 0 generation 0
a=candidate:2564284589 1 udp 1677730047 75.100.125.142 45834 typ srflx raddr 0.0.0.0 rport 0 generation 0
a=candidate:2422434839 1 udp 2113937663 10.0.3.122 63054 typ host generation 0
a=ice-options:ice2,trickle
a=mid:0
a=rtpmap:26 JPEG/90000
a=rtcp-fb:26 goog-remb
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=rtcp-mux
a=rtcp:9 IN IP4 0.0.0.0
a=end-of-candidates
a=sendonly
a=ssrc:1297181212 cname:d0860e88-a474-405d-a44f-e54a39390539
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:XSOI
a=ice-pwd:ETVICNGSOOBXHCHIXVYUUGDH
a=fingerprint:sha-256 A2:2C:21:02:5C:9E:A7:C7:47:E1:C9:ED:13:C5:6F:6E:96:71:21:C1:69:DB:9C:69:60:68:20:4A:4E:92:F2:BA
a=setup:passive
a=ice-options:ice2,trickle
a=mid:1
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=sctp-port:5000
a=max-message-size:262144

Remote peer receiving answer:

v=0
o=- 32058 0 IN IP4 127.0.0.1
s=sipsorcery
t=0 0
a=group:BUNDLE 0 1
m=video 9 UDP/TLS/RTP/SAVP 26
c=IN IP4 0.0.0.0
a=ice-ufrag:JRJJ
a=ice-pwd:DNVTQRUDPANEWTNZBVOJFZOC
a=fingerprint:sha-256 5D:1D:39:7C:FE:29:91:4E:C8:6C:CF:5B:55:D1:D2:BC:38:5E:3C:AF:79:41:4D:82:32:F0:6E:A4:E7:A1:DD:C6
a=setup:active
a=candidate:3913191201 1 udp 8447 157.245.114.91 59754 typ relay raddr 0.0.0.0 rport 0 generation 0
a=candidate:2564284589 1 udp 1677730047 75.100.125.142 52771 typ srflx raddr 0.0.0.0 rport 0 generation 0
a=candidate:2422434839 1 udp 2113937663 10.0.3.122 61986 typ host generation 0
a=ice-options:ice2,trickle
a=mid:0
a=rtpmap:26 JPEG/90000
a=rtcp-fb:26 goog-remb
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=rtcp-mux
a=rtcp:9 IN IP4 0.0.0.0
a=end-of-candidates
a=inactive
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:JRJJ
a=ice-pwd:DNVTQRUDPANEWTNZBVOJFZOC
a=fingerprint:sha-256 5D:1D:39:7C:FE:29:91:4E:C8:6C:CF:5B:55:D1:D2:BC:38:5E:3C:AF:79:41:4D:82:32:F0:6E:A4:E7:A1:DD:C6
a=setup:active
a=ice-options:ice2,trickle
a=mid:1
a=sctp-port:5000
a=max-message-size:262144

ChristopheI · 2024-09-10T06:46:34Z

Local peer receiving offer:

You mean instead "Local peer creates offer", isn't it ?
Then send it to Remote Peer. And you have shared here the SDP sent to Remote peer.

Remote peer receiving answer:

You mean instead "Remote peer creates answer using offer", isn't it ?
Then send it to Local Peer. And you have shared here the SDP sent to the Local peer.

Local peer receiving offer, with new video track:

Same as before: You mean instead "Local peer creates offer", isn't it ?
Then send it to Remote Peer. And you have shared here the SDP sent to Remote peer.

Remote peer receiving answer:

Again: You mean instead "Remote peer creates answer using offer", isn't it ?
Then send it to Local Peer. And you have shared here the SDP sent to the Local peer.

If all I assume before is correct,

it's the Local Peer which add Video and don't want to receive any because in the Offer SDP we have this;:

m=video 9 UDP/TLS/RTP/SAVP 26
...
a=sendonly

Remote Peer dont' want also to have Video because in its answer SDP we have this:

m=video 9 UDP/TLS/RTP/SAVP 26
...
a=inactive

evan-hilscher · 2024-09-10T06:53:59Z

No, I meant it the way I wrote it. I'm debugging both ends of the connection, and I was only logging the OfferReceived (called by the "local" peer) and AnswerReceived (called by the "remote" peer) methods.

The remote peer is adding the video, and the local peer is answering with inactive.

ChristopheI · 2024-09-10T06:58:27Z

Ok my bad.
So on Local Peer, once you have received the SDP offer with a new video track,. If you want to receive it, you have to add a Video track with RecvOnly mode

evan-hilscher · 2024-09-10T07:01:45Z

That makes sense, but what is the best way to detect that the SDP offer contains a new video track? I would expect that the RTPSession would fire some kind of event to let my application know that the SDP offer contains a new video track.

ChristopheI · 2024-09-10T07:11:58Z

Perhaps there is a better way, but I'm doing this :

using a temporary PeerConnection, I set the SDP as Remote description
I check then Audio/video Stream list and their state
According my needs, on the correct PeerConnection I add / update Audio/Video stream
then on the correct PeerConnection I set the SDP as Remote description

Extract

var temporaryPeerConnection = new RTCPeerConnection(rtcConfiguration, videoAsPrimary: videoAsPrimary);
temporaryPeerConnection.setRemoteDescription(rtcSessionDescriptionInit);
int nbAudio = temporaryPeerConnection.AudioStreamList.Count;
int nbVideo = temporaryPeerConnection.VideoStreamList.Count;

// Check status of Audio Remote Tracks
...

// Check status of Video Remote Tracks
...

var result = rtcPeerConnection.setRemoteDescription(rtcSessionDescriptionInit);

ChristopheI · 2024-09-10T07:17:13Z

Notice also I follow this rules:

I never change media order in the SDP
if a media is no more necessary, I set it to "inactive" (it's never removed for the SDP)

So I can manage this kind of scenario:

Audio communication, with up to 2 video in same time (web cam + screen sharing)
At any moment web cam and/or screen sharing on both peers can be added/"removed" (i.e. disabled)

evan-hilscher · 2024-09-10T07:21:32Z

I see. Interesting workaround, basically using a dummy RTCPeerConnection as a parser. My workaround was searching for inactive tracks after setting the remote description on the live connection, but I think it's fragile.

The WebRTC specification contains a "track" event (https://w3c.github.io/webrtc-pc/#event-track), which sipsorcery does not appear to implement. I am not certain, but I think the "track" event exists for just this situation.

evan-hilscher · 2024-09-10T07:22:55Z

Actually, I see it commented out here:

sipsorcery/src/net/WebRTC/IRTCPeerConnection.cs

Lines 412 to 421 in 06f8e3c

    
           // TODO: Extensions for the RTCMediaAPI 
        
           // https://www.w3.org/TR/webrtc/#rtcpeerconnection-interface-extensions. 
        
           //List<IRTCRtpSender> getSenders(); 
        
           //List<IRTCRtpReceiver> getReceivers(); 
        
           //List<RTCRtpTransceiver> getTransceivers(); 
        
           //RTCRtpSender addTrack(MediaStreamTrack track, param MediaStream[] streams); 
        
           //void removeTrack(RTCRtpSender sender); 
        
           //RTCRtpTransceiver addTransceiver((MediaStreamTrack or DOMString) trackOrKind, 
        
           ////optional RTCRtpTransceiverInit init = {}); 
        
           //event ontrack;

ChristopheI · 2024-09-10T07:31:32Z

Yes. Don't hesitate to create a PR ;)
There is so much that can be done to further improve features of this library...

evan-hilscher · 2024-09-10T07:39:17Z

Dang, I was afraid you'd say that 😜 Guess that's my week sorted.

sipsorcery · 2024-11-07T21:36:12Z

The whole SDP & media re-negotiation logic was never implemented (it was hard enough to get things working with a single audio and video track).

It's a sizeable job.

ha-ves mentioned this issue Sep 10, 2024

One-sided MediaStreamTrack always Inactive? (How to activate an Inactive track?) #1166

Closed

sipsorcery added enhancement webrtc sdp labels Nov 7, 2024

sipsorcery closed this as completed Nov 10, 2024

sipsorcery mentioned this issue Nov 10, 2024

Enhancements #1226

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot detect when a track is added by the remote peer #1135

Cannot detect when a track is added by the remote peer #1135

evan-hilscher commented Jun 27, 2024

ChristopheI commented Sep 4, 2024

evan-hilscher commented Sep 5, 2024

ChristopheI commented Sep 6, 2024

evan-hilscher commented Sep 9, 2024

ChristopheI commented Sep 10, 2024

evan-hilscher commented Sep 10, 2024

ChristopheI commented Sep 10, 2024

evan-hilscher commented Sep 10, 2024 •

edited

Loading

ChristopheI commented Sep 10, 2024

ChristopheI commented Sep 10, 2024

evan-hilscher commented Sep 10, 2024

evan-hilscher commented Sep 10, 2024

ChristopheI commented Sep 10, 2024

evan-hilscher commented Sep 10, 2024

sipsorcery commented Nov 7, 2024

Cannot detect when a track is added by the remote peer #1135

Cannot detect when a track is added by the remote peer #1135

Comments

evan-hilscher commented Jun 27, 2024

ChristopheI commented Sep 4, 2024

evan-hilscher commented Sep 5, 2024

ChristopheI commented Sep 6, 2024

evan-hilscher commented Sep 9, 2024

ChristopheI commented Sep 10, 2024

evan-hilscher commented Sep 10, 2024

ChristopheI commented Sep 10, 2024

evan-hilscher commented Sep 10, 2024 • edited Loading

ChristopheI commented Sep 10, 2024

ChristopheI commented Sep 10, 2024

evan-hilscher commented Sep 10, 2024

evan-hilscher commented Sep 10, 2024

ChristopheI commented Sep 10, 2024

evan-hilscher commented Sep 10, 2024

sipsorcery commented Nov 7, 2024

evan-hilscher commented Sep 10, 2024 •

edited

Loading