High RAM and CPU Usage, Broken OpenVINO Support, and Stability Issues with RTMP and go2rtc After Upgrade to 0.13.1

[VoidNullNil]

Issue Summary

After upgrading to version 0.13.1, we are experiencing multiple critical issues that significantly degrade the performance and functionality of our application. The problems include unexpectedly high RAM and CPU usage, apparent broken support for OpenVINO, RTMP functionality ceasing, and instability in go2rtc (versions 1.7.1 and up).

Detailed Description

• Increased Resource Usage: Post-upgrade, the application's RAM usage has more than doubled 500->1200, and CPU usage has also unexpectedly spiked. This increase is observed under normal operating conditions, without any changes to our usage patterns or configurations.

• OpenVINO Support: The integration with OpenVINO now seems to be broken, with processors showing much higher usage than expected. This suggests a possible issue with OpenVINO support or optimization in the latest version.

• RTMP and go2rtc Stability: RTMP functionality has stopped working entirely post-upgrade. Additionally, the go2rtc integration has become highly unstable, with issues beginning from version 1.7.1. The specific go2rtc issue is documented here: go2rtc issue #947.

Environment

• Software Version: 0.13.1
• Operating System: Ubuntu 22.04 LTS with OpenVino and VAAPI support
• Relevant Software Versions: OpenVINO, go2rtc 1.7.1 and above
• Hardware: CPU: Intel N5105, 16 Gigs of RAM

Expected Behavior

Prior to the upgrade, the application was running efficiently with manageable RAM and CPU usage. OpenVINO support was stable, and both RTMP and go2rtc functionalities were working as expected without any issues.

Actual Behavior

• Resource Usage: The application now consumes double the amount of RAM and significantly more CPU, impacting performance and scalability.
• OpenVINO Support: Integration with OpenVINO is no longer stable, leading to excessive CPU usage.
• RTMP and go2rtc: RTMP has ceased to function, and go2rtc stability issues have emerged, directly affecting our operational capabilities.

Steps to Reproduce

1. Upgrade to version 0.13.1.
2. Observe the increased RAM and CPU usage under normal operation.
3. Test OpenVINO integration and note the high processor usage.
4. Attempt to use RTMP and go2rtc functionalities, noting any failures or stability issues.

Request

We are seeking guidance on mitigating these issues or an update that addresses these critical problems. Any workarounds or patches that can temporarily alleviate these issues would also be greatly appreciated.

[NickM-27]

There is not much information to go off of here, I'd suggest providing your full config, logs, and a screenshot of the system page.

The release notes noted that motion detection changed and would be more sensitive by default potentially resulting in higher CPU usage for some users. Sounds like this may be the case and motion detection should be tuned https://docs.frigate.video/configuration/motion_detection

[NickM-27]

I looked at it. I just tested RTMP with my reolink camera and it is working normally so this is likely not an across the board rtmp issue

[NickM-27]

also worth noting that you can replace the included version of go2rtc with whatever version you want

https://docs.frigate.video/configuration/advanced#custom-go2rtc-version

[VoidNullNil]

Thank you, I have already determined the version that has issues, as detailed here: AlexxIT/go2rtc#947.

[NickM-27]

yes, but in that case you can use 1.7.1 and it will work for you without issues until go2rtc resolves the problem

[VoidNullNil]

No, CPU and RAM usage was the same (doubled) from 0.12.1.

[VoidNullNil]

I'd suggest providing your full config, logs, and a screenshot of the system page.

I'd suggest initiating an issue tracker and welcoming bug reports. It appears that the introduction of new features might be impacting the performance and functionality of existing ones.

[NickM-27]

https://docs.frigate.video/configuration/motion_detection

[VoidNullNil]

I don't have a server rack and can't afford to allocate CPU resources to detecting white noise.

[NickM-27]

The docs walk through step by step how to get motion detection to work optimally for your cameras. This will reduce cpu usage.

[VoidNullNil]

I provided you with the configuration; there is no motion detection whatsoever. It's fun. Who knows what those streams are detecting.

[NickM-27]

Right, so you should make motion detection less sensitive like I've said many times. For example, you could try disabling improve_contrast and see how it improves. The docs walk through step by step how to ensure excess motion is not detected but objects that you want to detect are detected even when there is low contrast like at night.

Otherwise, if you don't want to configure anything, you may be best off just using 0.12

I don't think there's any more support that I can offer at this point.

[VoidNullNil]

[NickM-27]

[VoidNullNil]

Does the term "discussion category" not resonate with you? I was inquiring about the section for issues.

[NickM-27]

we recently moved from issues to discussions as this is what GitHub recommends. Issues are for things that can be closed with a PR like feature requests.

[VoidNullNil]

Sure, feel free to submit a new PR request regarding this issue: AlexxIT/go2rtc#947.

[VoidNullNil]

Thanks, @NickM-27, for your patience with the responses in this thread. Your answers prompted me to conduct some tests. The first thing I noticed was a change in how motion detection works for all cameras. The parameters in detect are somewhat unclear compared to version 0.12. Specifying global parameters:

detect:
  # Optional: width of the frame for the input with the detect role (default: use native stream resolution)
  width: 1280
  # Optional: height of the frame for the input with the detect role (default: use native stream resolution)
  height: 720

partially reduced CPU usage. Memory usage increased, but I understand this is the cost for the new motion detection method. Considering all these changes, even OpenVINO struggles to keep up with tracking the changes. Therefore, one must pay attention to the parameter:

motion:
  # Optional: The threshold passed to cv2.threshold to determine if a pixel is different enough to be counted as motion. (default: shown below)
  # Increasing this value will make motion detection less sensitive and decreasing it will make motion detection more sensitive.
  # The value should be between 1 and 255.
  threshold: 25

If you have a mini PC, the only solution is to buy an Edge TPU and set up the correct model. The one that comes with Frigate for Edge TPU is about twice as bad as the one for OpenVINO.

I'll say this: if you have a lot of time, you can play around and even get good results. But for the average person, it's all too complicated. It probably also played a role that the backend uses Python, where solutions in, for example, Go/Rust would probably use half as many resources.

In any case, thanks to the team for your work and your contribution to open source. And once again, my apologies to @NickM-27 for my harsh messages.

[VoidNullNil]

Edge TPU model roughly equivalent in quality to the one for OpenVINO: SSD MobileNet V2 (300x300). You can find the model here: SSD MobileNet V2 Edge TPU version.