Add new --vid-stride inference parameter for videos

[VELCpro]

@glenn-jocher as seen in the issue #8079 (comment) there's the need sometimes to skip frames while performing inference on a video.
One of the useful use cases could be batch processing of very long archive videos where high frame rates are not needed.

I added two parameters to detect.py

• --inf-rate --> number of inference given a frame rate unit
• --fr-unit --> frame rate unit ('fps', 'fpm', 'fph')

Where :

• fps = frames per second
• fpm = frames per minute
• fph = frames per hour

example use:
python detect.py --device 0 --weights yolov5s.pt --inf-rate 1 --fr-unit fps --source vid.mp4

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Enhancement to video frame processing with the introduction of frame-rate stride control.

📊 Key Changes

• 🖼️ Added a vid_stride parameter to control the stride of video frame processing (skipping frames).
• 🛠️ vid_stride is implemented in LoadImages and LoadStreams classes, affecting how videos are loaded and processed.
• 🎛️ The vid_stride option is added as a command-line argument in both predict.py and detect.py scripts, permitting customization from the command-line interface.

🎯 Purpose & Impact

• 🚀 The purpose is to allow users to speed up video processing by analyzing fewer frames (skipping over frames based on stride value).
• ✨ This can significantly reduce processing time and resource usage when high frame-rate precision is not required.
• 🌐 It can also be beneficial for streaming scenarios where live feedback is more important than processing every single frame.

[github-actions]

👋 Hello @VELCpro, thank you for submitting a YOLOv5 🚀 PR! To allow your work to be integrated as seamlessly as possible, we advise you to:

• ✅ Verify your PR is up-to-date with ultralytics/yolov5 master branch. If your PR is behind you can update your code by clicking the 'Update branch' button or by running git pull and git merge master locally.
• ✅ Verify all YOLOv5 Continuous Integration (CI) checks are passing.
• ✅ Reduce changes to the absolute minimum required for your bug fix or feature addition. "It is not daily increase but daily decrease, hack away the unessential. The closer to the source, the less wastage there is." — Bruce Lee

[glenn-jocher]

@VELCpro thanks for the PR! The code additions are too complicated to be added to master, especially the multiple units, but there is an example in LoadStreams with a single variable called read that manages frame skipping that we could use as a starting point.

A couple things to remember here are that detect.py and classify/predict.py would both need to be updated, and that LoadStreams and LoadImages should share common arguments and funcitonalities, so if we exposed a new variable at the argparser it should affect both identically, i.e. --variable 30 to read at 1 frame out of every 30, default --variable 1

yolov5/utils/dataloaders.py

Lines 377 to 383 in 9da6d0f

	n, f, read = 0, self.frames[i], 1 # frame number, frame array, inference every 'read' frame
	while cap.isOpened() and n < f:
	n += 1
	# _, self.imgs[index] = cap.read()
	cap.grab()
	if n % read == 0:
	success, im = cap.retrieve()

[VELCpro]

@glenn-jocher thanks for the feedback!
I can try to simplify the code using only a sort of "skip_frame" variable

[glenn-jocher]

@VELCpro thanks for the updates, this looks a lot simpler now! Is this ready to merge? Have you tested on videos?

[VELCpro]

@glenn-jocher yes I tested on videos and on a folder with multiple videos and seems ok.

I didn't test the piece of code about rtsp stream but seems trivial. The change only added a variable, the logic behind is untouched

[glenn-jocher]

@VELCpro looks good now. The only issue I see is that the default naming and value make it seem like we are skipping every 1 frame, i.e. defaulting to 15 FPS. Not sure if there's a better wording here like vid_stride or something. What do you think?

[VELCpro]

@glenn-jocher yes actually naming the variable skip_frame could cause some confusion.

vid_stride seems ok to me

[glenn-jocher]

@VELCpro perfect. I'll update.

[glenn-jocher]

@VELCpro PR is merged. Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐

[VELCpro]

@glenn-jocher happy to help 🙂

[Mohammad-Saad-eng]

@VELCpro @glenn-jocher Hello, hope you are doing well. I have trained a custom model on yolov8n which will detect the night traffic. I have to use it on 8 streams (or 4 in worst case) but I am facing too much lag. And when I use vid_stride, it skips the frames but the stream is still very slow. Is there a way to get better fps for that much streams using yolov8n?

[VELCpro]

Hi @Mohammad-Saad-eng everything good thank you for asking.
What about you?

As you notice the vid_stride parameter allows to skip frames during the detection over a video.
This could be useful if its not needed to do detection on every single frame in real time or in doing bulk analysis on large video files.

Other ways to improve latency without touching the actual models are for example to reduce the input size of the frames given to the model for the detection.

Clearly it's a matter of trade-offs between latency and accuracy.
It's probable that you are hitting the hardware boundaries of your system with the specific yolo model selected.

What hardware you are testing the model on?
Have you tested the model on a single stream to validate the latency?

[Mohammad-Saad-eng]

@VELCpro I am also fine.

I have tested it on Jetson Nano so far. Jetson is currently using DeepStream and TrafficCamNet for 8 streams. As number of parameters in yolov8n is significantly smaller than TrafficCamNet, I figured yolo would work fine too.

[VELCpro]

Hi @Mohammad-Saad-eng

Me and my team are currently using a Jetson orin 32GB in conjunction with yolov8n to perform object tracking obtaining good results both in terms of accuracy and fps.

The Jetson Nano should be capable of running yolov5n (despite the limited power and the different architecture in respect to yolov8n)

Here you can find an fps benchmark analysis performed by seeed studio where yolov5n6 run FP16 inferences in 48ms on a Jetson Nano.

I think also they used directly the yolo docker container without deepstream.
To further enhance speed other than using vid_stride you can try TesorRT model format in FP16 or INT8 to speed up the computation at inference time. Here a benchmark of the speed gains with different model format and jeston devices

Hope it helps

[glenn-jocher]

Hi @VELCpro-eng,

We have successfully used a Jetson Orin 32GB with yolov8n for object tracking, achieving good results in terms of both accuracy and FPS.

Although the Jetson Nano has limited power and a different architecture compared to yolov8n, it should still be capable of running yolov5n. Seeed Studio has performed an FPS benchmark analysis, running yolov5n6 in FP16 inferences in 48ms on a Jetson Nano. You can find the analysis here.

In addition to using the vid_stride parameter, you can also try using the TensorRT model format in FP16 or INT8 to further enhance speed during inference. Seeed Studio has conducted a benchmark testing the speed gains with different model formats on Jetson devices, which you can find here.

I hope these suggestions help.