pure Rust RTSP client library

[scottlamb]

Drop ffmpeg's libavformat for RTSP. Use a pure Rust RTSP library instead. None seems to exist today, so probably write one.

Why?

• make building and installing easier. This is pretty much obsoleted by the Docker build.
• security. I don't like considering the cameras trusted—they're poorly written at best, hostile at worst. I'd like to interface with them using memory-safe Rust code rather than C to improve security.
• Address ffmpeg missing features / bugs. Timeouts (not supported on Raspbian Jessie at least and buggy even on modern ffmpeg, see RTSP/ffmpeg: incorrect "bad cseq" log message after socket timeout #126), audio problems (see poor behavior when camera has audio enabled #36), other buffering/pts oddities (https://trac.ffmpeg.org/ticket/5018). Some of the other errors folks have seen (mismatched CSeq and such) may or may not be due to ffmpeg problems; we haven't fully diagnosed them (see eg One of Four Cameras Is Failing #114).
• maybe add support for RTP/H.264 NAL unit types 25 (STAP-B), 26 (MTAP-16), 27 (MTAP-24), and 29 (FU-B). ffmpeg is missing these now, and one of my cameras occasionally logs errors about them as it drops the connection. I think it's very rare to actually use these; none of my cameras do. ffmpeg is just logging about them because the stream is getting corrupted either on the camera side or because of the timeout problem mentioned above. So I probably won't add this support unless someone actually needs it.
• allow easily accessing timestamps in the 0xABAC ONVIF RTP extension and RTCP sender reports.
• likely necessary for SVC support. Looks like no, at least with the "pseudo-SVC" that my Hikvision and Dahua cameras have. (Not standard SVC, just altering what H.264 frames are used as references, which with some work on my part could become temporal SVC.)
• be a bit more efficient (less copying, no transforming into an intermediate Annex B format), although it's surely insignificant compared to the cost of on-NVR analytics.
• future RTSP server/proxying support, as an alternative to Moonfire NVR's custom web socket/mp4 fragment live view protocol.

We'll still want ffmpeg's libavcodec for H.264 decoding for on-NVR video analytics, although that could happen in a separate process than the main moonfire-nvr one.

[skrzepto]

Looks like another repo created a working rust rtsp client/server here https://github.com/revmischa/cloudcam/pull/23/files which we can ask to pull out to a seperate library.

There's also https://github.com/sgodwincs/rtsp-rs but its incomplete at the moment.

[jlpoolen]

I'm coming to the mindset that ffmpeg when used as a client against my Reolink cameras has shortcomings. That brought me back to this issue.

Since you use ffmpeg as an externally created rtsp client, what about the possibility of using an alternative? Somewhere in my investigation I determined that Reolink is using Live555 code for their rtsp servers on the cameras I have. Live555 is open source and they have a client (I may try to build the client and then see how it fares with long open connections to my Reolink cameras) and an integrated client library with less bells and whistles.

I'm just wondering if the integration of ffmpeg into Moonfire-nvr was a Herculean effort that you do not want to undertake for another outside client. It looks like a pure rust version for rtsp V. 1 is a long way off as of 3/14/2021. I'm thinking the ffmpeg client may be the weak link and wondering if the Live555 might be more robust, especially at handling network interruptions, disconnects or other problems long extended connections (days, weeks, months) present.

Have you tried the Live555 client? If not, I'll do so. I do not want to repeat steps you have already taken and determined to be unproductive.

[scottlamb]

I haven't tried Live555. I don't know off-hand how difficult it'd be to use, but it'd be interesting anyway to see if Live555 is significantly more able than ffmpeg to maintain RTSP sessions with those cameras.

[scottlamb]

btw, I also think pure-Rust RTSP support might be closer than it appears. There are several interrelated pieces to replace the ffmpeg RTSP usage:

• extracting width and height from the H.264 SPS/PPS. Moonfire NVR already uses the h264-reader crate to get some other data from them; adding this part should be trivial.
• RTP, RTCP, and SDP: the recent webrtc.rs project has added crates for these that look pretty good. see webrtc-rtp, webrtc-rtcp, and webrtc-sdp. They're young, and focused on WebRTC, but they might actually have everything we need.
• RTSP itself. This is a relatively small piece.

[scottlamb]

I played around a bit here: scottlamb/moonfire-playground@b9ad194 It's indeed not too hard to do the basic stuff to get RTP data flowing.

[scottlamb]

I'm trying to get some API feedback here for a new RTSP crate. https://users.rust-lang.org/t/rtsp-streaming-api-sanity-check/59464

[jlpoolen]

yeah Team!  I'll try to spend some time this weekend... Scott has been busy and I need to catch up.

On 5/6/2021 1:55 PM, Scott Lamb wrote: I'm trying to get some API feedback here for a new RTSP crate. https://users.rust-lang.org/t/rtsp-streaming-api-sanity-check/59464 <https://users.rust-lang.org/t/rtsp-streaming-api-sanity-check/59464> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#37 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABXP4KVK2GCXIDWPOE22TDTML63JANCNFSM4EUPUNAA>.

-- Email Rider John Laurence Poole 1566 Court ST NE Salem OR 97301-4241 707-812-1323 office

[scottlamb]

btw, I also think pure-Rust RTSP support might be closer than it appears.

Well, I was pretty far off with this. There was a lot of work left to do (eg neither RTP library really had the codec depacketization stuff in the way I needed). But I'm making a pure Rust RTSP library anyway! I moved it from the playground spot mentioned above to its own repository: https://github.com/scottlamb/retina Lots of unfinished stuff there but it basically works.

I haven't integrated it into Moonfire NVR yet. It expects to run on the tokio threads, and those sometimes stall in Moonfire NVR now due to #88 , so I might fix that first. And then I might make a flag for my pure Rust RTSP library vs ffmpeg rather than commit to it right away. And then perhaps on to the audio support...

[jlpoolen]

On 6/4/2021 7:38 AM, Scott Lamb wrote: btw, I also think pure-Rust RTSP support might be closer than it appears. Well, I was pretty far off with this. There was a lot of work left to do (eg neither RTP library really had the codec depacketization stuff in the way I needed). But I'm making a pure Rust RTSP library anyway! I moved it from the playground spot mentioned above to its own repository: https://github.com/scottlamb/retina <https://github.com/scottlamb/retina> Lots of unfinished stuff there but it basically works. I haven't integrated it into Moonfire NVR yet. It expects to run on the tokio threads, and those sometimes stall in Moonfire NVR now due to #88 <#88> , so I might fix that first. And then I might make a flag for my pure Rust RTSP library vs ffmpeg rather than commit to it right away. And then perhaps on to the audio support... — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#37 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABXP4MLL2AWT5DNHIGWP6LTRDQP7ANCNFSM4EUPUNAA>.

Here's my build log and notes for those who do not want to think, but just do it: Quick Steps on How to Build and run the example client Note: requires rust version 1.52 or higher [from /usr/local/src where "git clone [URL]" is performed] cd retina cargo build [ There will be aproximately 169 crates.] To see help:  cargo run --example client help To pull an MP4 file from a Reolink camera  cargo run --example client mp4 test_output.mp4 --url rtsp://192.168.1.48:554/h264Preview_01_main --username [user] --password [secret] creates an outputfile of 17M after running for several seconds then Ctrl-C to terminate process. MP4 has AUDIO!! Yeah Scott! -- Email Rider John Laurence Poole 1566 Court ST NE Salem OR 97301-4241 707-812-1323 office

[scottlamb]

Moonfire NVR 0.6.4 uses the new RTSP crate by default. I'll wait a while for bug reports before ripping out ffmpeg support. Once I fully commit to the new library, I'll likely rework the threading model (no more need for a thread per stream).

[jlpoolen]

Bravo.