Added frame-skipping tolerance parameter

[cbteeple]

Package/Feature:

image_view, video_recorder

Description:

Added a tolerance parameter for frame-skipping that enables the user to choose how much variation in framerate is acceptable.

What this fixes:

This is a fix for Issue #604. Adding a frame-skipping tolerance allows us to change the condition for frame skipping to allow for some natural variation (which often occurs when streaming video from a webcam using usb_cam for example).

Details:

The tolerance parameter is a percent, and is passed to the video_recorder node via the parameter server:

<node name="video_recorder_node" pkg="image_view" type="video_recorder" respawn="false" output="screen">
    <remap from="image" to="/usb_cam/image_raw"/>
    <param name="fps" value="30" />
    <param name="tolerance" value="15" />
    <param name="codec" value="avc1" />
    <param name="filename" value="output.mp4" />
</node>

The condition in video_recorder for frame skipping changes from:

if ( (image_msg->header.stamp - g_last_wrote_time) < ros::Duration(1.0 / fps) )

to

if ( (image_msg->header.stamp - g_last_wrote_time) < ros::Duration( (1.0 / fps)*(1.0-tolerance/100) ) )

This allows some amount of variation in the framerate to be acceptable.

Testing

Platform: Ubuntu 20.04, ROS Noetic
Hardware: Logitech C920 Webcam

1. Start a camera stream using usb_cam, with 30fps using the mjpeg format to get the direct compressed stream from this camera.
2. Start a video_recorder with 30 fps, 15% tolerance, and the avc1 codec (for H.264 compression)
3. No frames are dropped!
4. Repeat this with 0% tolerance: approximately every 5th frame dropped, so video is 20% shorter than it should be, and plays back too fast.

I think this confirms that the addition of the frame-skip tolerance has fixed the problem.

[JWhitleyWork]

Would you mind making this a value from 0-1 instead? Additionally, I'd prefer something more descriptive to indicate that it's a tolerance for the FPS rate (e.g. fps_tolerance_pct).

[cbteeple]

I agree with switching it to 0-1. Then the "tolerance" can be interpreted as a fraction of the frame duration (1/fps). In that case, can I call the parameter something like fps_tolerance_fraction or maybe fps_variation_allowance?

[JWhitleyWork]

Since this is likely to happen often on some systems, I would prefer this be a _THROTTLE at minimum and a _DEBUG if you're OK with it instead of an _INFO. Terminal spam is not recommended.

[cbteeple]

I understand the concern over terminal spam, so I'd be fine with setting this to _DEBUG.

I'm not sure what you mean by "throttle at minimum"? Do you mean I should change the debug print statement to say "Throttle at maximum framerate" or something like that?

[JWhitleyWork]

Sorry, no, I meant using one of the _THROTTLE macros so the output of this message is throttled to once per time t (see http://wiki.ros.org/roscpp/Overview/Logging - the Throttle section).

[cbteeple]

Oh nice! That's super handy. I didn't know the _THROTTLE type existed. Okay, so then what's a reasonable period? Once per second? Before, there was no notice of skipped frames at all. Do you think once every 2 seconds is fine?

Another thing: I'm not sure if it matters but I also changed the way frames are counted here (before, the skipped frames were not counted in g_count. But I felt it made sense for g_count to be "every frame that has ever arrived" rather than just "every frame that was saved". Thoughts on this?

[JWhitleyWork]

Ping @cbteeple for changes based on feedback.

[clalancette]

There's been no response to the review on this PR for over a year, so I'm going to close this out. If you'd still like to get this in, feel free to reopen and respond to feedback.

[RoboCrafty]

I’ve implemented this tolerance and even tried setting a tire like 50% and 30% but there is still a lot of frame skipping. At times a five minute video turns out to be 2 minutes because of skipping.

My launch file:

 <launch>
    <node pkg="image_view" type="video_recorder" name="video_recorder0" args="_image_transport:=compressed"> 
        <remap from="image" to="/zed_node/rgb/image_rect_color" />
        <param name="filename" value="/home/nvidia/surveillance/ZED-Cam_$(arg timestamp).mp4" />
			<param name="codec" value="avc1" />
    		<param name="fps" value="11" />
    		<param name="tolerance" value="50" />
    </node>
 
    <node pkg="image_view" type="video_recorder" name="video_recorder3" args="_image_transport:=compressed">
        <remap from="image" to="/usb_cam_03/image" />
			<param name="codec" value="avc1" />
        <param name="filename" value="/home/nvidia/surveillance/Back-Cam_$(arg timestamp).mp4" />
    		<param name="fps" value="20" />
    		<param name="tolerance" value="50" />
    </node>
</launch>