Stream Freezes and Skips Randomly on Home Assistant Companion App

[thinkjk]

On my local network on the Android and iOS home assistant companion app my stream freezes after a few seconds and I can’t get it to unfreeze without navigating away on the app. Sometimes it’ll freeze, then jump forward a bunch of seconds and freeze again. It works perfectly in my browser.

It only happens with WebRTC. RTMP, RTSP, and JSMpeg play without any issues.

Here’s my current config:

Camera → RTSP → Frigate → RTMP → Home Assistant. I have RTSPtoWebRTC pointing to go2rtc. My go2rtc.yaml is empty. I see the RTMP stream added automatically in the go2rtc UI when I start a stream.

I see the freezing issue with the Frigate Lovelace Card as well as the Picture Glace card.

Troubleshooting I’ve done:
If I remove go2rtc and RTSPtoWebRTC and add a generic camera with a direct RTSP from the camera it does not freeze.
If I add the RTMP stream from Frigate the stream does no freeze.
I tried using your older WebRTC plugin only and I get the same issue, stream freezes when it flips to WebRTC (I know it’s webrtc because I’ve verified in developer tools and the icon flipped to webrtc in the corner)

Additional testing. I have 3 types of Amcrest cameras and it seems like the issue is mainly happening on one model the ip4m-1051. Is there anything I can do to troubleshoot or resolve the issue?

Thanks.

[AlexxIT]

You can share your camera stream with me, so I can check it with debugger. You can do it using Ngrok tunnel as example.

[thinkjk]

Sure! Thanks @AlexxIT. Can you quickly let me know how I can test the stream externally? I have the ngrok client installed on home assistant from https://github.com/dy1io/hassio-addons. I added the ngrok.yaml and updated my go2rtc.yaml to point to the config (per your docs). I opened up the WebRTC and API port on my firewall, but I'm not sure how to test if I can connect externally.

Edit: I get an error in the go2rtc logs saying I'm limited to one connection ERR_NGROK_108. I think this is because I was having the client from the add-on and go2rtc try to make a connection. I removed the api key from the add-on while keeping it only in the ngrok.yaml file so go2rtc can use it. I still can't figure out how to test the stream externally.

Can I have a friend try to hit the stream somehow with VLC?

[AlexxIT]

https://github.com/AlexxIT/go2rtc/wiki/Tunnel-RTSP-camera-to-Intenet

You can send link to contacts from my github page

[thinkjk]

@AlexxIT I sent you a friend request on Discord unless you prefer Telegram (which I've never used)

[thinkjk]

Leaving a final note. Thanks to @AlexxIT for the troubleshooting help. This was due to 2 issues.

1. The ip4m-1051 streams in very very high quality, so lower the quality helped with the issue a bit (though it was still dropping frames.).
2. The fix ended up being that there was added latency coming from Frigate (RTMP). I had to add a generic camera (RTSP) and then I could stream without issues or lag.

[rogerquake]

After I switched to the direct RTSP feeds from my cameras and not the Frigate RTMP feeds, I am not seeing any freezing issues either.

[thinkjk]

@AlexxIT this issue started happening again on the newer versions of the app. @rogerquake are you running into this as well running the newest version?

[rogerquake]

Yeah, my streams have frozen twice on the latest release.

[AlexxIT]

Well. Beta9 has this problem, but Beta8 doesn't?

[rogerquake]

I don't think it's any different honestly. Beta8 froze also, so don't think Beta9 is any worse.

[AlexxIT]

Well. Which version was definitely OK?

[thinkjk]

Beta5 was my last confirmed working version for me. I unfortunately don't have a backup of that version but I just tested Beta6 and get the same freezing issue. There were also updated with the home assistant mobile app since that time.

[NickM-27]

Beta5 was my last confirmed working version for me. I unfortunately don't have a backup of that version but I just tested Beta6 and get the same freezing issue. There were also updated with the home assistant mobile app since that time.

You can load previous releases in the releases section. https://github.com/AlexxIT/go2rtc/releases/tag/v0.1-beta.5

[AlexxIT]

If you put any go2rtc binary to Hass config folder - it will be used by the add-on. Don't forget to set chmod +x

[thinkjk]

I just tried this and it's still showing beta9 in the UI. I have go2rtc_linux_amd64 in my config folder and ran a chmod +x on it and rebooted HA.

[AlexxIT]

Binary should be named just go2rtc. Version in the UI won't change.

[thinkjk]

If you put any go2rtc binary to Hass config folder - it will be used by the add-on. Don't forget to set chmod +x

Thanks @AlexxIT. I just confirmed that with beta5 I don't have any issues at all, but every version after that my stream freezes and jumps 10+ seconds at a time. I tested with 6-9

[AlexxIT]

I need to access your stream again.

[NickM-27]

I'm seeing this with my streams while viewing webRTC in the browser. At least in my case it seems only the video stutters. I can hear up to date audio even when the frame for video is frozen.

[AlexxIT]

It's important to provide full info about source of your stream. Technology, transcoding, brands.

[NickM-27]

I've seen this on my amcrest ad410 which is h.264 / pcma. This one uses the direct rtsp link from the camera

I've also seen this on my reolink 511-WA using their http:// link with ffmpeg: preset. The stream is native h.264 and aac and I am using #video=copy#audio=opus to transcode the audio so it is usable.

[NickM-27]

I realized maybe this is more of a client issue? Watching with webrtc on my windows or mac desktop in chrome the stream does not skip at all.

However, watching in chrome on my android does see the stuttering quite often. Watching in chrome on my iPhone does not see the suttering.

[thinkjk]

I realized maybe this is more of a client issue? Watching with webrtc on my windows or mac desktop in chrome the stream does not skip at all.

However, watching in chrome on my android does see the stuttering quite often. Watching in chrome on my iPhone does not see the suttering.

If you read my original issue it's the same for me. I have 0 issues on desktop in any browser (Chrome, Firefox, Safari) and OS (Windows, OSX, Ubuntu). My issue only happens on mobile (I'm using the HA Companion App)

[NickM-27]

Okay thanks, glad we're seeing the same thing.

I wonder if there's some chrome flag or other thing that could improve it on Android. My phone has a Snapdragon 8+ Gen 1 so I don't think it's a performance issue.

[AlexxIT]

@NickM-27 can you check SDP for both cameras? You can increase log level to rtsp: trace or use ffplay -v trace from PC. This line:

a=fmtp:96 packetization-mode=1;profile-level-id=420032;sprop-parameter-sets=...

[thinkjk]

Okay thanks, glad we're seeing the same thing.

I wonder if there's some chrome flag or other thing that could improve it on Android. My phone has a Snapdragon 8+ Gen 1 so I don't think it's a performance issue.

Also if you have a minute do you mind trying to see if beta5 gives you the same issues on mobile? That version gives me no problems. @AlexxIT today my stream was freezing again on beta9 but works fine on beta5.

[NickM-27]

@AlexxIT

Amcrest AD410:
a=fmtp:96 packetization-mode=1;profile-level-id=420032;sprop-parameter-sets=Z0IAMukAUAHjQgAAB9IAAOqcCAA=,aMqPIAA=

Reolink 511-WA:
a=fmtp:96 packetization-mode=1; sprop-parameter-sets=J2QAKq0AzoB4AiflmoCAgPgAAAMACAAAAwDxgYABDzwADLbv/+BQ,KO48sA==; profile-level-id=64002A

[NickM-27]

@thinkjk I just tried beta 5 and I see the stuttering there as well.

[AlexxIT]

@thinkjk try beta.10

[thinkjk]

@thinkjk try beta.10

Thank you, it's much better than before! But I now see this weird issue where it shows the first frame of the video but takes 10-30+ seconds before it actually starts showing the live feed. It takes longer externally than internally. I currently have this in my go2rtc.yaml. Is there something else I should be changing for better connection?

webrtc:
  listen: ":8555"  # address of your local server (TCP)
  candidates:
    - MYIP:8555  # if you have static public IP-address
    ```

[AlexxIT]

Your camera sends very very rare keyframe. You can start watch stream only from keyframe. I think sometimes connection take time longer than first keyframe and you just miss it. In the future I can cache it and send again. But now have other tasks to do.

[thinkjk]

Thanks again @AlexxIT! Do you want me to submit an enhancement issue for that? Otherwise I'll close this issue :)

[AlexxIT]

If Stream Freezes and Skips Randomly fixed - you may close this issue and open new about caching keyframe.

[brendann993]

I appear to have the same issue specifcally on mobile(iOS).
Both in the Companion App & Saffari browser. Basically stream plays for about a second and then freezes for about a second and plays again then freezes again, etc.
On a PC steam appears smoth with no freezing.
SDP from camera:
a=fmtp:96 packetization-mode=1;profile-level-id=420029;sprop-parameter-sets=Z01AMpWgCiA9flmwEA==,aO48gA==
Using 0.1-beta.10
Device: iPhone 14 Pro
Device OS: iOS 16
Using the older WebRTC card provided smooth playback.

[AlexxIT]

@brendann993 what camera model do you use? This is direct RTSP or ffmpeg?

[thinkjk]

@AlexxIT do you want me to close this issue or keep it open so other people can post similar problems in? What would be easier for you to keep track of?

[brendann993]

@AlexxIT Its a generic Swann/Raysharp 1080p IP camera. Direct RTSP stream, Ill try with a few other IP cameras I have and some generic RTSP streams to see if its specific to Camera or not

[brendann993]

Just tested with a online RTSP test stream rtsp://wowzaec2demo.streamlock.net:554/vod/mp4:BigBuckBunny_115k.mp4
and its the issue is there as well so I don't think its camera specific.

[AlexxIT]

@brendann993 thanks for good example of a very bad stream. Your link is not a real stream. Which sends packets in irregular groups. Look carefully at: the time of the log, the time stamp in the frame and the sequence number of the frame.
That's why the stream plays "ugly". Real cameras don't do that.

I don't know if it's worth fixing. Because a camera like this is just garbage. You can create a separate issue about this problem if you want.

@thinkjk you can close this issue. Other people can continue the discussion either in a closed issue or open a new one.

07:56:48.280 DBG [RTP] codec: H264, nalu: 24, size:   1014, ts:       7470, pt: 97, ssrc: 1708923310, seq: 1, true

07:56:48.468 DBG [RTP] codec: H264, nalu:  1, size:    163, ts:      15030, pt: 97, ssrc: 1708923310, seq: 2, true
07:56:48.476 DBG [RTP] codec: H264, nalu:  1, size:     13, ts:      11250, pt: 97, ssrc: 1708923310, seq: 3, true
07:56:48.477 DBG [RTP] codec: H264, nalu:  1, size:    380, ts:      18720, pt: 97, ssrc: 1708923310, seq: 4, true
07:56:48.478 DBG [RTP] codec: H264, nalu:  1, size:   1331, ts:      26280, pt: 97, ssrc: 1708923310, seq: 5, true

07:56:48.611 DBG [RTP] codec: H264, nalu:  1, size:      8, ts:      22500, pt: 97, ssrc: 1708923310, seq: 6, true
07:56:48.615 DBG [RTP] codec: H264, nalu: 28, size:   1360, ts:      41220, pt: 97, ssrc: 1708923310, seq: 7, false
07:56:48.615 DBG [RTP] codec: H264, nalu: 28, size:     23, ts:      41220, pt: 97, ssrc: 1708923310, seq: 8, true
07:56:48.616 DBG [RTP] codec: H264, nalu:  1, size:     24, ts:      33750, pt: 97, ssrc: 1708923310, seq: 9, true

07:56:48.781 DBG [RTP] codec: H264, nalu:  1, size:     29, ts:      29970, pt: 97, ssrc: 1708923310, seq: 10, true
07:56:48.786 DBG [RTP] codec: H264, nalu:  1, size:     28, ts:      37530, pt: 97, ssrc: 1708923310, seq: 11, true
07:56:48.786 DBG [RTP] codec: H264, nalu:  1, size:     83, ts:      48780, pt: 97, ssrc: 1708923310, seq: 12, true
07:56:48.787 DBG [RTP] codec: H264, nalu:  1, size:     27, ts:      45000, pt: 97, ssrc: 1708923310, seq: 13, true
07:56:48.787 DBG [RTP] codec: H264, nalu:  1, size:     32, ts:      56250, pt: 97, ssrc: 1708923310, seq: 14, true
07:56:48.787 DBG [RTP] codec: H264, nalu:  1, size:     19, ts:      52470, pt: 97, ssrc: 1708923310, seq: 15, true

07:56:48.971 DBG [RTP] codec: H264, nalu:  1, size:    132, ts:      63720, pt: 97, ssrc: 1708923310, seq: 16, true
07:56:48.971 DBG [RTP] codec: H264, nalu:  1, size:     24, ts:      60030, pt: 97, ssrc: 1708923310, seq: 17, true

07:56:49.139 DBG [RTP] codec: H264, nalu:  1, size:    158, ts:      71280, pt: 97, ssrc: 1708923310, seq: 18, true
07:56:49.139 DBG [RTP] codec: H264, nalu:  1, size:     37, ts:      67500, pt: 97, ssrc: 1708923310, seq: 19, true
07:56:49.139 DBG [RTP] codec: H264, nalu:  1, size:    256, ts:      78750, pt: 97, ssrc: 1708923310, seq: 20, true
07:56:49.139 DBG [RTP] codec: H264, nalu:  1, size:     40, ts:      74970, pt: 97, ssrc: 1708923310, seq: 21, true
07:56:49.139 DBG [RTP] codec: H264, nalu:  1, size:    652, ts:      86220, pt: 97, ssrc: 1708923310, seq: 22, true
07:56:49.139 DBG [RTP] codec: H264, nalu:  1, size:     60, ts:      82530, pt: 97, ssrc: 1708923310, seq: 23, true

[brendann993]

This a example of the freezing experience. Top feed is the WebRTC card, bottom is the Picture Entity card streaming via go2rtc. Same experience in the app and browser on iOS.
I understand if its a bad camera at fault but why does the WebRTC card work fine but go2rtc doesnt? and why only in iOS, on a PC the stream in both cards is fine
https://user-images.githubusercontent.com/6630349/199837677-57786cd0-e5ae-4ff8-83de-550fbc31607b.mp4

[brendann993]

Just to wrap this one all up, I just confirmed the issue definetly comes from the camera.
I just added the camera in question to a RSTP relay that I have setup as my other camera only supports 1 stream and then added the relays output to home assistant and then stream is now smooth so obvious the direct connection between camera to go2rtc has some issues on the camera side.
So camera directly to homeassistant/go2rtc = stream freezes every couple of seconds on iOS
Camera to RSTP relay to homeassistant/go2rtc = smooth stream

[thinkjk]

So camera directly to homeassistant/go2rtc = stream freezes every couple of seconds on iOS Camera to RSTP relay to homeassistant/go2rtc = smooth stream

What relay are you using?

[brendann993]

https://github.com/aler9/rtsp-simple-server
I don't fully understand why the stream via the relay would be any better being as its just taking the stream in and republishing it but it works with no noticable delay for me at at least. Running WebRTC card (direct from camera) and HomeAssistant picture card (Via relay to go2rtc) someone walking through frame happens at the exact same time.

My other camera with stream limitations is also terrable so that steam runs through ffmpeg first to just drop any corrupt packs and then the output is restreamed.

[AlexxIT]

@brendann993 maybe your stream has some other problems. Not like in Bunny stream. You can share your stream with me. I'll check what I can do.

[thinkjk]

I wanted to leave a final comment on this issue. There was a new firmware released at the end of 2022 for my cameras that have totally fixed this issue. If you're running into the same issue please update your camera firmware.

[ShivaSVK]

I had the same problem with new Hikvision cameras. The RTC stream using the "custom:webrtc-camera" worked well on laptop browser, smartphone, but not on my Android tablet. For the first few seconds, while it was MSE, it worked fine. Skipping and freezing started after the stream changed to RTC. My solution is to add "mode: mse" line to the yaml configuration of the card. Now it works well on all devices (there is 1-2 second lag of the video, but still better than freezing video).