[IPCamera] No FFMPEG Motion Detection

[rueckix]

Motion detection via ffmpeg is not working reliably in low time resolution environments.

Expected Behavior

Motion detection triggered upon scene change.

Current Behavior

No motion detected regardless of scene changes and configured motion threshold.

Possible Solution

Looking at the code, I suspect that counting motion indicators (scene changes "lavfi.scene") and status reports ("speed=") is not working correctly. ffmpeg, by default, appears to be reporting "speed=" every second. Therefore any detected scene changes will immediately be cancelled out.

Alternatives

• Trigger motion alarm per each (except the first) detected scene. Then, it all depends on the threshold configuration
• Keeping time and counting the number of changed scenes per time interval. ==> less intuitive for the user because there are two tuning parameters (sensitivity and time frame)
• Increasing the reference period without explicit time keeping. I.e., counting the number of scene changes per, say, 10 status ("speed=") reports.

Steps to Reproduce (for Bugs)

1. Set up an mjpeg stream with a time resolution of 1Hz or slower.
2. Ingest in ipcamera and configure motion detection (threshold != 0)
3. Enabled debug logs for ipcamera binding
4. Watch the logs

Context

Detect motion in a stream of 1 picture per second coming off an ESP32-cam.

Your Environment

• On docker (latest), x64
• ffmpeg version 4.1.6-1~deb10u1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 8 (Debian 8.3.0-6)

[Skinah]

Thanks for reporting as this is something I did not consider that someone would have a 1 Frame per second stream. The way I have setup an esp32 cam for testing, I get a higher frame rate but I may have had to drop the resolution down from the max to achieve this.

Your correct about the speed= resets the counter that triggers motion, so it will probably be an issue for anyone that has <4FPS. It may also be an issue for people that use very high frame rates as I use 6 to 10 frames per second here.

The below is going from memory as I have not played with this in > 1 year now, so if your testing shows otherwise then please feel free to correct me...

The problem I was trying to solve was detecting motion of say a person that stands still and then waves only a hand and arm at the camera when they are standing back and are already a small size in the cameras FOV. If the size of the hand and arm is <5% of the cameras field of view it becomes very difficult to reliably detect it as motion if you use the suggested fix of triggering every time the threshold is crossed. I tried this first and it did not work very well.

My guess as to why this is difficult, is that in low light conditions you get noise across the full picture as you get a snow like noise in the picture, when this is happening it is very difficult to detect a small object moving over this noise. It may also be moving grass in the wind or a clock ticking down in the picture. This meant that to get reliable detection the size of the moving object had to be big enough to 'overpower' the background noise. The way the coding is done now is the camera has to see motion spread over a greater time frame then just a single frame and it actually works pretty well even with a clock ticking down in the footage. You can detect a small object moving over the noise when I played with it last.

I do believe it would be possible to fix this and still keep a single control for the user to tweak. Does FFmpeg list the frame rate when it opens the stream? If so it could be used to work out how many frames are in a second and use that in a formula.

[Skinah]

I did some testing and found that ffmpeg was very unreliable at detecting the FPS on a mjpeg stream coming from 2 different sources so that idea does not work. Since your using an esp32 camera I can not use the future planned RTSP functions to supply the FPS of the stream so this leaves 1 option that is easy to implement.

When the control is set to 100 it will then detect motion on every instance of lavfi.

Can you test this by using DEBUG mode of the logs? I found it needed to be a threshold of 100 for it to be reliable with a 1FPS feed in lower amounts of light.

I tested the camera with:
25FPS and needed a threshold of 9
20FPS and needed a threshold of 20
10FPS and needed a threshold of 18
5FPS and needed a threshold of 3

I'll have a think about what other options may be possible.

[rueckix]

Sorry, I was not watching this last week.

I need to find a bit of time to reconfigure my OH instance accordingly. I changed my setup quite a bit in the meantime.
Now, I am not certain about the threshold 100 setting. Could you explain why that should work?

Note for others with the same issue: I moved my config to external motion detection triggered by Motion. Upon motion detection, Motion calls a script that published a flag via MQTT. OH is watching and connects it to the external motion channel of IPCamera. This works reliably but of course has some overhead. Also, Motion's detection is based on pixel changes, rather than mpeg'ish scene discovery. That leads to false positives.

[Skinah]

Threshold of 100 will not currently work at changing the items state, however by watching the logs you can see if the line lavfi. only comes up when you have motion. From my testing even 90 gave false positives purely due to the low light level noise that a cameras sensor has.

I believe I have identified a few things that need to be considered.

1. When high frame rates are used, any motion that occurs will be a smaller pixel change as the frames are closer together in time.
2. With lower frame rates like 1FPS the pixel change is greater as someone has walked a great distance in the 1 second VS how far they would have walked in 1/30th of a second in a 30FPS rate.
3. Because movement is greater between frames in a 1FPS stream the threshold control needs to be much higher and this can allow a single value to trigger the motion alarm.
4. With high frame rates you need to get multiple frames to agree that there is motion when using lower threshold values.

I can probably trial a new algorithm that has a wider threshold range and will automatically decrease the motionCount counter required when the threshold is higher. Long term if the binding can get the FPS from onvif or RTSP routines this variable can be automatic, but for a camera like yours this is not possible without increasing CPU load and complexity.

I would recommend using motion above doing this in the binding as it is easier to draw areas to ignore motion in and really motion is doing a very similar thing to the binding. Some programs use a smarter pixel change detection that makes sure the pixels are grouped closely together and not spread over the whole cameras FOV so they will always work better. Also I would look at doing object detection with the 1 FPS stream, there are some forum posts about doing this and will lower the false positives, as will adding in a PIR motion sensor which are cheap and do not require large amounts of CPU to be chewed up.

[Skinah]

OK I have made some changes that I need tested, if your interested in helping you can download a jar from http://pcmus.com/openhab/IpCameraBinding/
Uninstall the binding and then just drop the jar into the addons folder and everything should just work the same with 1 change being the following...

Threshold now controls two variables, one being the amount of pixels that have to change. The higher the number the more pixels need to change. The second is best thought of as how many frames need to agree that motion is occurring. The smaller the change your trying to detect the more frames need to agree and this is probably only going to work on streams that are 3 FPS and higher. I did add the ability for a single frame to trigger motion when the motion control is 90 and above. So this is where you will want to test with your 1 FPS stream. I will be doing testing here over a range of different cameras and frame rates to see how the changes work before doing a PR so your feedback would be great to see if the settings work for you.

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/how-to-turn-a-cameras-rtsp-stream-into-motion-detection/89906/15

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/ipcamera-new-ip-camera-binding/42771/2749

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/how-to-handle-motion-detection-between-the-esp32-cam-and-openhab/144801/4