Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update MOT17 metrics and add MOT20 results #202

Merged
merged 4 commits into from
Oct 14, 2022
Merged

Update MOT17 metrics and add MOT20 results #202

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9622537
Added camera motion tracking to motmetrics demo
facundo-lezama Oct 6, 2022
a46b900
Added MOT20 to motmetrics demo.
facundo-lezama Oct 6, 2022
1b7bfba
Simplify motmetrics test
facundo-lezama Oct 7, 2022
637b60b
Added MOT20 and updated MOT17 metrics in README
facundo-lezama Oct 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 21 additions & 26 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,32 +162,27 @@ If you are looking for a tracker, here are some other projects worth noting:

## Benchmarks

[MOT17](https://motchallenge.net/data/MOT17/) results obtained using [motmetrics4norfair](https://github.com/tryolabs/norfair/tree/master/demos/motmetrics4norfair) demo script. Hyperparameters were tuned for reaching a high `MOTA` on this dataset. A more balanced set of hyperparameters, like the default ones used in the other demos, is recommended for production.

| | Rcll | Prcn | GT MT PT ML | FP FN IDs FM | MOTA MOTP |
| :------------: | :---: | :---: | :--------------: | :--------------------: | :---------: |
| MOT17-13-DPM | 18.5% | 85.8% | 110 6 31 73 | 355 9490 90 100 | 14.7% 26.7% |
| MOT17-04-FRCNN | 56.3% | 93.2% | 83 18 43 22 | 1959 20775 90 104 | 52.0% 10.7% |
| MOT17-11-FRCNN | 61.4% | 92.9% | 75 18 34 23 | 443 3639 65 62 | 56.1% 10.2% |
| MOT17-04-SDP | 77.5% | 97.4% | 83 49 25 9 | 1003 10680 232 257 | 74.9% 13.2% |
| MOT17-13-SDP | 57.6% | 83.3% | 110 46 26 38 | 1349 4934 164 163 | 44.6% 20.0% |
| MOT17-05-DPM | 38.1% | 82.9% | 133 11 58 64 | 544 4281 90 100 | 28.9% 24.3% |
| MOT17-09-DPM | 59.7% | 75.1% | 26 4 17 5 | 1052 2147 120 117 | 37.7% 26.3% |
| MOT17-10-DPM | 37.5% | 84.9% | 57 6 19 32 | 856 8024 127 153 | 29.8% 24.9% |
| MOT17-02-SDP | 50.9% | 75.9% | 62 11 38 13 | 3003 9122 272 290 | 33.3% 18.4% |
| MOT17-11-DPM | 54.2% | 84.7% | 75 12 24 39 | 927 4321 87 63 | 43.5% 21.7% |
| MOT17-09-FRCNN | 58.5% | 98.5% | 26 7 17 2 | 49 2209 40 39 | 56.8% 9.6% |
| MOT17-11-SDP | 75.7% | 90.9% | 75 34 30 11 | 719 2297 112 105 | 66.9% 14.1% |
| MOT17-02-FRCNN | 36.5% | 79.5% | 62 7 26 29 | 1751 11796 124 136 | 26.4% 13.4% |
| MOT17-05-FRCNN | 54.9% | 90.0% | 133 23 69 41 | 420 3123 78 84 | 47.7% 18.0% |
| MOT17-04-DPM | 42.5% | 83.5% | 83 7 44 32 | 3985 27335 406 432 | 33.2% 21.1% |
| MOT17-10-SDP | 74.1% | 88.0% | 57 30 24 3 | 1295 3323 283 286 | 61.8% 19.8% |
| MOT17-10-FRCNN | 60.9% | 75.7% | 57 14 37 6 | 2507 5020 310 306 | 39.0% 17.2% |
| MOT17-09-SDP | 67.7% | 94.6% | 26 12 14 0 | 204 1722 54 56 | 62.8% 13.0% |
| MOT17-02-DPM | 20.2% | 81.4% | 62 5 14 43 | 856 14833 113 109 | 15.0% 24.6% |
| MOT17-13-FRCNN | 59.3% | 74.7% | 110 33 53 24 | 2334 4735 342 331 | 36.3% 18.4% |
| MOT17-05-SDP | 66.8% | 87.9% | 133 32 81 20 | 637 2299 134 133 | 55.6% 16.5% |
| OVERALL | 53.7% | 87.3% | 1638 385 724 529 | 26248 156135 3333 3426 | 44.9% 16.4% |
[MOT17](https://motchallenge.net/data/MOT17/) and [MOT20](https://motchallenge.net/data/MOT17/) results obtained using [motmetrics4norfair](https://github.com/tryolabs/norfair/tree/master/demos/motmetrics4norfair) demo script on the `train` split. We used detections obtained with [ByteTrack's](https://github.com/ifzhang/ByteTrack) YOLOX object detection model.

| MOT17 Train | IDF1 IDP IDR | Rcll | Prcn | MOTA MOTP |
| :---------: | :---------------: | :---: | :---: | :---------: |
| MOT17-02 | 61.3% 63.6% 59.0% | 86.8% | 93.5% | 79.9% 14.8% |
| MOT17-04 | 93.3% 93.6% 93.0% | 98.6% | 99.3% | 97.9% 07.9% |
| MOT17-05 | 77.8% 77.7% 77.8% | 85.9% | 85.8% | 71.2% 14.7% |
| MOT17-09 | 65.0% 67.4% 62.9% | 90.3% | 96.8% | 86.8% 12.2% |
| MOT17-10 | 70.2% 72.5% 68.1% | 87.3% | 93.0% | 80.1% 18.7% |
| MOT17-11 | 80.2% 80.5% 80.0% | 93.0% | 93.6% | 86.4% 11.3% |
| MOT17-13 | 79.0% 79.6% 78.4% | 90.6% | 92.0% | 82.4% 16.6% |
| OVERALL | 80.6% 81.8% 79.6% | 92.9% | 95.5% | 88.1% 11.9% |


| MOT20 Train | IDF1 IDP IDR | Rcll | Prcn | MOTA MOTP |
| :------: | :---------------: | :---: | :---: | :---------: |
| MOT20-01 | 85.9% 88.1% 83.8% | 93.4% | 98.2% | 91.5% 12.6% |
| MOT20-02 | 72.8% 74.6% 71.0% | 93.2% | 97.9% | 91.0% 12.7% |
| MOT20-03 | 93.0% 94.1% 92.0% | 96.1% | 98.3% | 94.4% 13.7% |
| MOT20-05 | 87.9% 88.9% 87.0% | 96.0% | 98.1% | 94.1% 13.0% |
| OVERALL | 87.3% 88.4% 86.2% | 95.6% | 98.1% | 93.7% 13.2% |

## Commercial support

Expand Down
19 changes: 15 additions & 4 deletions demos/motmetrics4norfair/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,22 @@ RUN apt-get update && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*

# Download the MOT17 dataset
WORKDIR /
# To download Detections + Ground Truth (9.7 MB)
RUN curl -O https://motchallenge.net/data/MOT17Labels.zip && \
unzip MOT17Labels.zip -d MOT17
# Download the MOT17 dataset
RUN curl -O https://motchallenge.net/data/MOT17.zip && \
unzip MOT17.zip
RUN rm -rf MOT17/test

# Download the MOT20 dataset
RUN curl -O https://motchallenge.net/data/MOT20.zip && \
unzip MOT20.zip
RUN rm -rf MOT20/test

# Replace MOT Challenge's detections for ByteTrack's detections
RUN pip install gdown && \
gdown --folder https://drive.google.com/drive/folders/1m8alcbgt3e3XAWCq-iOQknGFKqFwSwt0?usp=sharing
RUN cp -r bytetrack_detections/MOT17/train/ /MOT17/
RUN cp -r bytetrack_detections/MOT20/train/ /MOT20/

WORKDIR /demo
COPY requirements.txt requirements.txt
Expand Down
9 changes: 1 addition & 8 deletions demos/motmetrics4norfair/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ What does each script do?
## Instructions

1. Build and run the Docker container with `./run.sh`.
2. Within the container, run `python motmetrics4norfair.py /MOT17/train/`.
2. Within the container, run `python motmetrics4norfair.py /MOT17/train/` to evaluate on `MOT17` dataset or `python motmetrics4norfair.py /MOT20/train/` to evaluate on `MOT20` dataset.
3. For more advanced use cases, within the container you can display the motmetrics4norfair instructions (replace `motmetrics4norfair.py` with `motmetrics4norfair_xyah.py` or `motmetrics4bytetrack.py` as needed):

```bash
Expand All @@ -24,13 +24,6 @@ or run the following for a quick test
python motmetrics4norfair.py /MOT17/train/
```

To download Detections + Ground Truth + Images (5.5GB)

```bash
curl -O https://motchallenge.net/data/MOT17.zip
unzip MOT17.zip
```

## Important consideration

Hyperparameters were tuned for reaching a high `MOTA` on this dataset. They may not be ideal for more general use cases. Id switches suffer especially due to this optimization. If you want to improve Id switches, use a higher `hit_counter_max`, or just use the default hyperparameters.
101 changes: 43 additions & 58 deletions demos/motmetrics4norfair/src/motmetrics4norfair.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,29 @@

import numpy as np

from norfair import Tracker, drawing, metrics, video
from norfair import drawing, metrics, Tracker, video
from norfair.camera_motion import MotionEstimator
from norfair.distances import iou
from norfair.filter import FilterPyKalmanFilterFactory

frame_skip_period = 1
detection_threshold = 0.01
distance_threshold = 0.9
diagonal_proportion_threshold = 1 / 18
pointwise_hit_counter_max = 3
hit_counter_max = 2
DETECTION_THRESHOLD = 0.5
DISTANCE_THRESHOLD = 0.9
POINTWISE_HIT_COUNTER_MAX = 3
HIT_COUNTER_MAX = 8


def build_mask(frame, detections, tracked_objects):
# create a mask of ones
mask = np.ones(frame.shape[:2], frame.dtype)
# set to 0 on detections and tracked_objects
for det in detections:
i = det.points.astype(int)
mask[i[0, 1] : i[1, 1], i[0, 0] : i[1, 0]] = 0
for obj in tracked_objects:
i = obj.estimate.astype(int)
mask[i[0, 1] : i[1, 1], i[0, 0] : i[1, 0]] = 0
return mask


parser = argparse.ArgumentParser(
description="Evaluate a basic tracker on MOTChallenge data. Display on terminal the MOTChallenge metrics results "
Expand Down Expand Up @@ -67,39 +81,6 @@

accumulator = metrics.Accumulators()


def keypoints_distance(detected_pose, tracked_pose):
norm_orders = [1, 2, np.inf]
distances = 0
diagonal = 0

hor_min_pt = min(detected_pose.points[:, 0])
hor_max_pt = max(detected_pose.points[:, 0])
ver_min_pt = min(detected_pose.points[:, 1])
ver_max_pt = max(detected_pose.points[:, 1])

# Set keypoint_dist_threshold based on object size, and calculate
# distance between detections and tracker estimations
for p in norm_orders:
distances += np.linalg.norm(
detected_pose.points - tracked_pose.estimate, ord=p, axis=1
)
diagonal += np.linalg.norm(
[hor_max_pt - hor_min_pt, ver_max_pt - ver_min_pt], ord=p
)

distances = distances / len(norm_orders)

keypoint_dist_threshold = diagonal * diagonal_proportion_threshold

match_num = np.count_nonzero(
(distances < keypoint_dist_threshold)
* (detected_pose.scores > detection_threshold)
* (tracked_pose.last_detection.scores > detection_threshold)
)
return 1 / (1 + match_num)


for input_path in sequences_paths:
# Search vertical resolution in seqinfo.ini
seqinfo_path = os.path.join(input_path, "seqinfo.ini")
Expand All @@ -114,35 +95,39 @@ def keypoints_distance(detected_pose, tracked_pose):
input_path=input_path, save_path=output_path, information_file=info_file
)

if args.make_video:
video_file = video.VideoFromFrames(
input_path=input_path, save_path=output_path, information_file=info_file
)
video_file = video.VideoFromFrames(
input_path=input_path,
save_path=output_path,
information_file=info_file,
make_video=args.make_video,
)

tracker = Tracker(
distance_function=keypoints_distance,
distance_threshold=distance_threshold,
detection_threshold=detection_threshold,
pointwise_hit_counter_max=pointwise_hit_counter_max,
hit_counter_max=hit_counter_max,
distance_function=iou,
distance_threshold=DISTANCE_THRESHOLD,
detection_threshold=DETECTION_THRESHOLD,
pointwise_hit_counter_max=POINTWISE_HIT_COUNTER_MAX,
hit_counter_max=HIT_COUNTER_MAX,
filter_factory=FilterPyKalmanFilterFactory(),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we still using filterpy? OptimizedFilter is faster and it shouldn't affect metrics too much.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried using OptimizedKalmanFilterFactory, but the metrics were a bit worse. We saw this when implementing OptimizedKalmanFilterFactory but we still haven't gotten to the bottom of it.

)

motion_estimator = MotionEstimator(max_points=500)

# Initialize accumulator for this video
accumulator.create_accumulator(input_path=input_path, information_file=info_file)

for frame_number, detections in enumerate(all_detections):
if frame_number % frame_skip_period == 0:
tracked_objects = tracker.update(
detections=detections, period=frame_skip_period
)
else:
detections = []
tracked_objects = tracker.update()
tracked_objects = []
for frame, detections in zip(video_file, all_detections):
mask = build_mask(frame, detections, tracked_objects)
coord_transformations = motion_estimator.update(frame, mask)

tracked_objects = tracker.update(
detections=detections,
coord_transformations=coord_transformations,
)

# Draw detection and tracked object boxes on frame
if args.make_video:
frame = next(video_file)
frame = drawing.draw_boxes(frame, detections=detections)
frame = drawing.draw_tracked_boxes(frame=frame, objects=tracked_objects)
video_file.update(frame=frame)
Expand Down
38 changes: 20 additions & 18 deletions norfair/video.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -300,37 +300,39 @@ def abbreviate_description(self, description: str) -> str:


class VideoFromFrames:
def __init__(self, input_path, save_path=".", information_file=None):
def __init__(
self, input_path, save_path=".", information_file=None, make_video=True
):

if information_file is None:
information_file = metrics.InformationFile(
file_path=os.path.join(input_path, "seqinfo.ini")
)
if make_video:
file_name = os.path.split(input_path)[1]

file_name = os.path.split(input_path)[1]
# Search framerate on seqinfo.ini
fps = information_file.search(variable_name="frameRate")

# Search framerate on seqinfo.ini
fps = information_file.search(variable_name="frameRate")
# Search resolution in seqinfo.ini
horizontal_resolution = information_file.search(variable_name="imWidth")
vertical_resolution = information_file.search(variable_name="imHeight")
image_size = (horizontal_resolution, vertical_resolution)

# Search resolution in seqinfo.ini
horizontal_resolution = information_file.search(variable_name="imWidth")
vertical_resolution = information_file.search(variable_name="imHeight")
image_size = (horizontal_resolution, vertical_resolution)
videos_folder = os.path.join(save_path, "videos")
if not os.path.exists(videos_folder):
os.makedirs(videos_folder)

# Search total frames in seqinfo.ini
self.length = information_file.search(variable_name="seqLength")
video_path = os.path.join(videos_folder, file_name + ".mp4")
fourcc = cv2.VideoWriter_fourcc(*"mp4v")

videos_folder = os.path.join(save_path, "videos")
if not os.path.exists(videos_folder):
os.makedirs(videos_folder)
self.file_name = file_name
# Video file
self.video = cv2.VideoWriter(video_path, fourcc, fps, image_size)

video_path = os.path.join(videos_folder, file_name + ".mp4")
fourcc = cv2.VideoWriter_fourcc(*"mp4v")

self.file_name = file_name
self.length = information_file.search(variable_name="seqLength")
self.input_path = input_path
self.frame_number = 1
self.video = cv2.VideoWriter(video_path, fourcc, fps, image_size) # Video file
self.image_extension = information_file.search("imExt")
self.image_directory = information_file.search("imDir")

Expand Down
46 changes: 23 additions & 23 deletions tests/metrics.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,23 +1,23 @@
IDF1 IDP IDR Rcll Prcn GT MT PT ML FP FN IDs FM MOTA MOTP IDt IDa IDm
MOT17-10-DPM 27.4% 44.8% 19.8% 37.5% 84.9% 57 6 19 32 856 8024 127 153 29.8% 0.249 19 116 9
MOT17-04-SDP 66.2% 74.6% 59.4% 77.5% 97.4% 83 49 25 9 1003 10680 232 257 74.9% 0.132 94 136 4
MOT17-11-SDP 55.2% 60.8% 50.6% 75.7% 90.9% 75 34 30 11 719 2297 112 105 66.9% 0.141 47 73 13
MOT17-13-FRCNN 48.1% 54.3% 43.1% 59.3% 74.7% 110 33 53 24 2334 4735 342 331 36.3% 0.184 115 232 17
MOT17-05-FRCNN 55.2% 72.9% 44.4% 54.9% 90.0% 133 23 69 41 420 3123 78 84 47.7% 0.180 39 55 16
MOT17-02-SDP 34.4% 42.8% 28.7% 50.9% 75.9% 62 11 38 13 3003 9122 272 290 33.3% 0.184 72 203 7
MOT17-13-SDP 56.0% 68.5% 47.4% 57.6% 83.3% 110 46 26 38 1349 4934 164 163 44.6% 0.200 47 126 21
MOT17-10-SDP 47.5% 51.9% 43.7% 74.1% 88.0% 57 30 24 3 1295 3323 283 286 61.8% 0.198 119 157 5
MOT17-02-FRCNN 33.9% 53.8% 24.7% 36.5% 79.5% 62 7 26 29 1751 11796 124 136 26.4% 0.134 37 95 9
MOT17-13-DPM 22.1% 62.3% 13.4% 18.5% 85.8% 110 6 31 73 355 9490 90 100 14.7% 0.267 14 81 5
MOT17-04-FRCNN 55.7% 73.9% 44.7% 56.3% 93.2% 83 18 43 22 1959 20775 90 104 52.0% 0.107 5 88 3
MOT17-04-DPM 29.2% 43.3% 22.0% 42.5% 83.5% 83 7 44 32 3985 27365 406 432 33.2% 0.211 48 352 1
MOT17-09-FRCNN 54.0% 72.4% 43.1% 58.5% 98.5% 26 7 17 2 49 2209 40 39 56.8% 0.096 15 29 4
MOT17-11-DPM 39.2% 50.3% 32.2% 54.2% 84.7% 75 12 24 39 927 4321 87 63 43.5% 0.217 20 68 4
MOT17-09-DPM 33.4% 37.7% 30.0% 59.7% 75.1% 26 4 17 5 1052 2147 120 117 37.7% 0.263 41 82 3
MOT17-05-SDP 58.1% 67.3% 51.1% 66.8% 87.9% 133 32 81 20 637 2299 134 133 55.6% 0.165 51 98 18
MOT17-09-SDP 53.2% 63.8% 45.6% 67.7% 94.6% 26 12 14 0 204 1722 54 56 62.8% 0.130 22 38 6
MOT17-11-FRCNN 54.3% 68.3% 45.1% 61.4% 92.9% 75 18 34 23 443 3639 65 62 56.1% 0.102 16 54 6
MOT17-10-FRCNN 39.8% 44.7% 35.9% 60.9% 75.7% 57 14 37 6 2507 5020 310 306 39.0% 0.172 122 186 9
MOT17-05-DPM 36.5% 57.9% 26.6% 38.1% 82.9% 133 11 58 64 544 4281 90 100 28.9% 0.243 33 68 13
MOT17-02-DPM 16.2% 40.9% 10.1% 20.2% 81.4% 62 5 14 43 856 14833 113 109 15.0% 0.246 20 92 1
OVERALL 45.4% 59.7% 36.7% 53.7% 87.3% 1638 385 724 529 26248 156135 3333 3426 44.9% 0.164 996 2429 174
IDF1 IDP IDR Rcll Prcn GT MT PT ML FP FN IDs FM MOTA MOTP IDt IDa IDm
MOT17-02-DPM 21.3% 36.6% 15.0% 26.7% 65.2% 62 6 18 38 2649 13620 155 149 11.6% 0.250 22 133 3
MOT17-11-FRCNN 53.9% 68.7% 44.3% 60.9% 94.5% 75 19 33 23 337 3690 62 58 56.7% 0.100 17 51 7
MOT17-13-SDP 61.0% 76.1% 50.9% 58.0% 86.7% 110 48 25 37 1035 4890 119 131 48.1% 0.196 48 82 24
MOT17-10-DPM 25.0% 29.9% 21.5% 42.9% 59.7% 57 8 21 28 3724 7327 185 224 12.5% 0.259 42 145 8
MOT17-09-DPM 34.3% 31.8% 37.2% 64.2% 55.0% 26 7 16 3 2795 1906 102 99 9.8% 0.262 31 73 4
MOT17-02-SDP 36.6% 46.7% 30.1% 49.6% 77.0% 62 11 36 15 2758 9369 253 308 33.4% 0.181 78 178 8
MOT17-04-DPM 30.1% 32.8% 27.8% 49.7% 58.6% 83 10 46 27 16725 23932 407 432 13.7% 0.224 69 335 4
MOT17-10-FRCNN 41.2% 47.4% 36.4% 60.4% 78.5% 57 14 37 6 2123 5086 270 301 41.7% 0.170 113 160 11
MOT17-04-FRCNN 55.5% 74.0% 44.4% 56.0% 93.3% 83 18 43 22 1920 20941 94 108 51.7% 0.106 10 87 3
MOT17-11-DPM 38.7% 40.2% 37.3% 57.8% 62.3% 75 14 27 34 3296 3979 84 84 22.0% 0.216 20 68 6
MOT17-09-FRCNN 50.6% 68.9% 40.0% 57.4% 99.0% 26 7 17 2 32 2266 34 35 56.2% 0.093 12 26 4
MOT17-10-SDP 47.9% 53.2% 43.5% 74.1% 90.5% 57 29 25 3 994 3323 242 256 64.5% 0.198 112 121 6
MOT17-05-SDP 60.0% 71.8% 51.5% 65.6% 91.5% 133 30 82 21 419 2379 110 125 58.0% 0.164 61 69 22
MOT17-04-SDP 68.4% 77.4% 61.2% 77.1% 97.5% 83 47 26 10 953 10911 183 233 74.7% 0.131 57 131 7
MOT17-09-SDP 51.1% 62.3% 43.3% 66.5% 95.7% 26 10 15 1 159 1782 51 50 62.6% 0.125 21 35 5
MOT17-13-FRCNN 51.2% 59.8% 44.8% 59.6% 79.5% 110 34 52 24 1783 4707 285 296 41.8% 0.183 108 193 18
MOT17-11-SDP 58.3% 65.2% 52.8% 75.0% 92.6% 75 32 32 11 564 2362 90 99 68.0% 0.139 45 59 14
MOT17-02-FRCNN 35.0% 56.7% 25.3% 36.0% 80.7% 62 7 26 29 1599 11893 113 127 26.8% 0.126 35 84 7
MOT17-05-DPM 34.6% 43.8% 28.6% 44.0% 67.3% 133 14 65 54 1478 3872 129 121 20.8% 0.253 55 89 17
MOT17-05-FRCNN 52.5% 71.9% 41.3% 53.6% 93.3% 133 25 66 42 265 3210 66 78 48.8% 0.181 49 43 26
MOT17-13-DPM 26.4% 41.7% 19.3% 25.5% 54.9% 110 10 36 64 2430 8678 176 212 3.1% 0.283 47 140 12
OVERALL 45.6% 55.6% 38.7% 55.4% 79.5% 1638 400 744 494 48038 150123 3210 3526 40.2% 0.167 1052 2302 216
Loading