From 0edc1eaa645fef8b41ebc6b19e2e141241600c14 Mon Sep 17 00:00:00 2001 From: javiber Date: Wed, 19 Oct 2022 09:59:25 -0300 Subject: [PATCH 1/2] Tabbed documentation and getting started --- docs/getting_started.md | 86 +++++++++++++++++++++++++++++++++++++++++ docs/reference/index.md | 9 ----- mkdocs.yml | 5 +++ norfair/__init__.py | 16 ++++++++ 4 files changed, 107 insertions(+), 9 deletions(-) create mode 100644 docs/getting_started.md diff --git a/docs/getting_started.md b/docs/getting_started.md new file mode 100644 index 00000000..2e1e463a --- /dev/null +++ b/docs/getting_started.md @@ -0,0 +1,86 @@ +# Getting Started + +Norfair's goal is to easily track multiple objects in videos based on the frame-by-frame detections of a user-defined model. + +## Model or Detector + +We recommend first deciding and setting up the model and then adding Norfair on top of it. +Models trained for any form of [object detection](https://paperswithcode.com/task/object-detection) or [keypoint detection](https://paperswithcode.com/task/keypoint-detection) (including [pose estimation](https://paperswithcode.com/task/pose-estimation)) are all supported. You can check some of the integrations we have as examples: + +- [Yolov7](https://github.com/tryolabs/norfair/tree/master/demos/yolov7), [Yolov5](https://github.com/tryolabs/norfair/tree/master/demos/yolov5) and [Yolov4](https://github.com/tryolabs/norfair/tree/master/demos/yolov4) +- [Detectron2](https://github.com/tryolabs/norfair/tree/master/demos/detectron2) +- [Alphapose](https://github.com/tryolabs/norfair/tree/master/demos/alphapose) +- [Openpose](https://github.com/tryolabs/norfair/tree/master/demos/openpose) +- [MMDetection](https://github.com/tryolabs/norfair/tree/master/demos/mmdetection) + +Any other model trained on one of the supported tasks is also supported and should be easy to integrate with Norfair, regardless of whether it uses Pytorch, TensorFlow, or other. + +If you are unsure of which model to use, [Yolov7](https://github.com/WongKinYiu/yolov7) is a good starting point since it's easy to set up and offers models of different sizes pre-trained on object detection and pose estimation. + +!!! Note + Norfair is a Detection-Based-Tracker (DBT) and as such, its performance is highly dependent on the performance of the model of choice. + +The detections from the model will need to be wrapped in an instance of [Detection][norfair.tracker.Detection] before passing them to Norfair. + +## Install + +Installing Norfair is extremely easy, simply run `pip install norfair` to install the latest version from [pypi](https://pypi.org/project/norfair/). + +You can also install the latest version from the master branch using `pip install git+https://github.com/tryolabs/norfair.git@master#egg=norfair` + +## Video + +Norfair offers optional functionality to process videos (mp4 and mov formats are supported) or capture a live feed from a camera. +To use this functionality you need to install Norfair with the `video` extra using this command: `pip install norfair[video]`. + +Check the [Video class][norfair.video.Video] for more info on how to use it. + +## Tracking + +Let's dive right into a simple example in the following snippet: + +``` python +from norfair import Detection, Tracker, Video, draw_tracked_objects + +detector = MyDetector() # Set up a detector +video = Video(input_path="video.mp4") +tracker = Tracker(distance_function="frobenious", distance_threshold=100) + +for frame in video: + detections = detector(frame) + norfair_detections = [Detection(points) for points in detections] + tracked_objects = tracker.update(detections=norfair_detections) + draw_tracked_objects(frame, tracked_objects) + video.write(frame) +``` + +The tracker is created and then the detections are fed to it one frame at a time in order. This method is called _online tracking_ and allows Norfair to be used in live feeds and real-time scenarios where future frames are not available. + +Norfair includes functionality for creating an output video with drawings which is useful for evaluating and debugging. We usually start with this simple setup and move from there. + +### Next Steps + +The next steps depend a lot on your goal and the result of evaluating the output videos, nevertheless here are some pointers that might help you solve common problems + +#### Detection Issues + +Most common problem is that the tracking has errors or is not precise enough. In this case, the first thing to check is whether this is a detection error or a tracking error. As mentioned above if the detector fails the tracking will suffer. + +To debug this use [`draw_points`][norfair.drawing.draw_points] or [`draw_boxes`][norfair.drawing.draw_boxes] to inspect the detections and analyze if they are precise enough. If you are filtering the detections based on scores, this is a good time to tweak the threshold. If you decide that the detections are not good enough you can try a different architecture, a bigger version of the model, or consider fine-tuning the model on your domain. + + +#### Tracking Issues + +After inspecting the detections you might find issues with the tracking, several things can go wrong with tracking but here is a list of common errors and things to try: + +- Objects take **too long to start**, this can have multiple causes: + - `initialization_delay` is too big on the Tracker. Makes the TrackedObject stay on initializing for too long, `3` is usually a good value to start with. + - `distance_threshold` is too big on the Tracker. Prevents the Detections to be matched with the correct TrackedObject. The best value depends on the distance used. + - Incorrect `distance_function` on the Tracker. Some distances might not be valid in some cases, for instance, if using IoU but the objects in your video move so quickly that there is never an overlap between the detections of consecutive frames. Try different distances, `frobenious` or `create_normalized_mean_euclidean_distance` are good starting points. +- Objects take **too long to disappear**. Lower `hit_counter_max` on the Tracker. +- Points or bounding boxes **jitter too much**. Increase `R` (measurement error) or lower `Q` (estimate or process error) on the `OptimizedKalmanFilterFactory` or `FilterPyKalmanFilterFactory`. This makes the Kalman Filter put less weight on the measurements and trust more on the estimate, stabilizing the result. +- **Camera motion** confuses the Tracker. If the camera moves, the apparent movement of objects can become too erratic for the Tracker. Use `MotionEstimator`. +- **Incorrect matches** between Detections and TrackedObjects, a couple of scenarios can cause this: + - `distance_threshold` is too big so the Tracker matches Detections to TrackedObjects that are simply too far. Lower the threshold until you fix the error, the correct value will depend on the distance function that you're using. + - Mismatches when objects overlap. In this case, tracking becomes more challenging, usually, the quality of the detection degrades causing one of the objects to be missed or creating a single big detection that includes both objects. On top of the detection issues, the tracker needs to decide which detection should be matched to which TrackedObject which can be error-prone if only considering spatial information. The solution is not easy but incorporating the notion of the appearance similarity based on some kind of embedding to your distance_finction can help. +- Can't **recover** an object **after oclussions**. Use ReID distance, see [this demo](TODO) for an example but for real-world use you will need a good ReID model that can provide good embeddings. diff --git a/docs/reference/index.md b/docs/reference/index.md index 65ca5050..c68d14b5 100644 --- a/docs/reference/index.md +++ b/docs/reference/index.md @@ -1,12 +1,3 @@ # Reference ::: norfair - -- [tracker][norfair.tracker] -- [drawing][norfair.drawing] -- [video][norfair.video] -- [distances][norfair.distances] -- [camera motion][norfair.camera_motion] -- [metrics][norfair.metrics] -- [filter][norfair.filter] -- [utils][norfair.utils] \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index dc9f542b..6923ffdd 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -6,6 +6,7 @@ theme: - navigation.instant - navigation.tracking - navigation.sections + - navigation.tabs icon: repo: fontawesome/brands/git-alt logo: img/logo-dark.svg @@ -20,8 +21,12 @@ extra_javascript: - js/math.js - https://polyfill.io/v3/polyfill.min.js?features=es6 - https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js + nav: + - Home: index.md + - Getting Started: getting_started.md - Reference: + - reference/index.md - Tracker: reference/tracker.md - Drawing: reference/drawing.md - Video: reference/video.md diff --git a/norfair/__init__.py b/norfair/__init__.py index abdca8c4..62d3050c 100644 --- a/norfair/__init__.py +++ b/norfair/__init__.py @@ -1,3 +1,19 @@ +""" +A customizable lightweight Python library for real-time multi-object tracking. + +Examples +-------- +>>> from norfair import Detection, Tracker, Video, draw_tracked_objects +>>> detector = MyDetector() # Set up a detector +>>> video = Video(input_path="video.mp4") +>>> tracker = Tracker(distance_function="frobenious", distance_threshold=50) +>>> for frame in video: +>>> detections = detector(frame) +>>> norfair_detections = [Detection(points) for points in detections] +>>> tracked_objects = tracker.update(detections=norfair_detections) +>>> draw_tracked_objects(frame, tracked_objects) +>>> video.write(frame) +""" import sys from .distances import * From d282efd3fa5f148b36b21356a50da70e3d817145 Mon Sep 17 00:00:00 2001 From: javiber Date: Wed, 26 Oct 2022 15:21:59 -0300 Subject: [PATCH 2/2] Getting started improvements Based on Facundo's feedback --- docs/getting_started.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/getting_started.md b/docs/getting_started.md index 2e1e463a..8bd5730a 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -24,7 +24,7 @@ The detections from the model will need to be wrapped in an instance of [Detecti ## Install -Installing Norfair is extremely easy, simply run `pip install norfair` to install the latest version from [pypi](https://pypi.org/project/norfair/). +Installing Norfair is extremely easy, simply run `pip install norfair` to install the latest version from [PyPI](https://pypi.org/project/norfair/). You can also install the latest version from the master branch using `pip install git+https://github.com/tryolabs/norfair.git@master#egg=norfair` @@ -44,7 +44,7 @@ from norfair import Detection, Tracker, Video, draw_tracked_objects detector = MyDetector() # Set up a detector video = Video(input_path="video.mp4") -tracker = Tracker(distance_function="frobenious", distance_threshold=100) +tracker = Tracker(distance_function="frobenius", distance_threshold=100) for frame in video: detections = detector(frame) @@ -75,12 +75,12 @@ After inspecting the detections you might find issues with the tracking, several - Objects take **too long to start**, this can have multiple causes: - `initialization_delay` is too big on the Tracker. Makes the TrackedObject stay on initializing for too long, `3` is usually a good value to start with. - - `distance_threshold` is too big on the Tracker. Prevents the Detections to be matched with the correct TrackedObject. The best value depends on the distance used. - - Incorrect `distance_function` on the Tracker. Some distances might not be valid in some cases, for instance, if using IoU but the objects in your video move so quickly that there is never an overlap between the detections of consecutive frames. Try different distances, `frobenious` or `create_normalized_mean_euclidean_distance` are good starting points. + - `distance_threshold` is too small on the Tracker. Prevents the Detections to be matched with the correct TrackedObject. The best value depends on the distance used. + - Incorrect `distance_function` on the Tracker. Some distances might not be valid in some cases, for instance, if using IoU but the objects in your video move so quickly that there is never an overlap between the detections of consecutive frames. Try different distances, `frobenius` or `create_normalized_mean_euclidean_distance` are good starting points. - Objects take **too long to disappear**. Lower `hit_counter_max` on the Tracker. - Points or bounding boxes **jitter too much**. Increase `R` (measurement error) or lower `Q` (estimate or process error) on the `OptimizedKalmanFilterFactory` or `FilterPyKalmanFilterFactory`. This makes the Kalman Filter put less weight on the measurements and trust more on the estimate, stabilizing the result. - **Camera motion** confuses the Tracker. If the camera moves, the apparent movement of objects can become too erratic for the Tracker. Use `MotionEstimator`. - **Incorrect matches** between Detections and TrackedObjects, a couple of scenarios can cause this: - `distance_threshold` is too big so the Tracker matches Detections to TrackedObjects that are simply too far. Lower the threshold until you fix the error, the correct value will depend on the distance function that you're using. - Mismatches when objects overlap. In this case, tracking becomes more challenging, usually, the quality of the detection degrades causing one of the objects to be missed or creating a single big detection that includes both objects. On top of the detection issues, the tracker needs to decide which detection should be matched to which TrackedObject which can be error-prone if only considering spatial information. The solution is not easy but incorporating the notion of the appearance similarity based on some kind of embedding to your distance_finction can help. -- Can't **recover** an object **after oclussions**. Use ReID distance, see [this demo](TODO) for an example but for real-world use you will need a good ReID model that can provide good embeddings. +- Can't **recover** an object **after occlusions**. Use ReID distance, see [this demo](https://github.com/tryolabs/norfair/tree/master/demos/reid) for an example but for real-world use you will need a good ReID model that can provide good embeddings.