Increase ar_thr from 20 to 100 for better detection on slender (high aspect ratio) objects

[MrinalJain17]

FIxes #2521

Being able to customize aspect ratio threshold (ar_thr) is quite important for using YOLOv5 with any custom dataset having a different distribution of bounding boxes than COCO.

There have been several issues in the past indicating that it's often necessary to modify the default value of 20 for ar_thr. For instance: #4030 #2070 #2521

The workaround is to manually edit the default value, but it's not a viable solution if the user intends to keep up to date with the origin Yolov5 repository (risking git merge errors).

This PR aims to make ar_thr available as a hyperparameter, and therefore customizable via hyp.yaml files.

@glenn-jocher I've drafted the PR based on my understanding on when and where box_candidates() is used within the code base. Let me know if I missed something, or if this needs more work.

EDIT: There was a similar PR #2869 to fix this issue. It made all the parameters of box_candidates() available as hyperparams.

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Enhanced bounding box filtering criteria in data augmentation.

📊 Key Changes

• The ar_thr (aspect ratio threshold) parameter in box_candidates function has been changed from 20 to 100.

🎯 Purpose & Impact

• 🎭 Purpose: This adjustment fine-tunes the data augmentation process, allowing for more varied aspect ratios of bounding boxes.
• 🕹️ Impact: Could lead to better generalization of the YOLOv5 model by effectively training on a wider range of object shapes, potentially improving detection performance in real-world scenarios.

[glenn-jocher]

@MrinalJain17 thanks for the PR! Yes it's true this is an open issue. I'm not sure adding additional hyps is the best idea though. How about one of these two options:

• Increase default value, i.e. 20 -> 40, or 20 -> 80. This will allow more use cases to pass without encountering limits.
• Auto-solve a value for each dataset. Load all annotations on trainloader init, find max AR, assign to self.ar_thr, use this value.

[MrinalJain17]

Auto-solve a value for each dataset. Load all annotations on trainloader init, find max AR, assign to self.ar_thr, use this value.

@glenn-jocher Option 2 seems much better to me, as opposed to hard-coding a value. I'll update the PR soon.

[glenn-jocher]

@MrinalJain17 yes option 2 sounds pretty cool.

But wait, maybe this is not a good idea. In this workflow, in a dataset with many low AR objects, i.e. VOC, where things are mostly square, our augmented mosaic loader will crop these annotations randomly, and this candidate criteria is in place as a threshold to help reject unwanted crops. i.e. if a vehicle is 90% cropped in one direction it's AR will multiply by 10, exceeding the un-augmented dataset AR stats, but we may still want to retain this cropped vehicle for training.

It's not clear what the best route is. VOC is easily trainable, we could run a study training it at various ar_thr and simply increase the default threshold.

[MrinalJain17]

@glenn-jocher That's an interesting point.

Although, I'd still argue that having a fixed ar_thr based on experiments from one specific dataset is not generalizable.

My proposal would be to have one of the following:

1. If we decide to hard-code ar_thr, then the user should have some way to customize it. Either through hyp.yaml files or as an argument to train.py (or any other alternative which does not involve manually modifying the function definition of box_candidates()).
   • The downside as you indicated earlier is one extra (hyper)parameter.
2. Or, we implement the auto-solve approach.
   • The downside here would be (potentially) unintended behavior with mosaic.

My point here is that the user manually modifying the threshold is problematic (and is necessary quite often it seems). In fact, I'm working on something right now where one of the object categories in my dataset had a non-existent recall, and it was because it had really high aspect ratios (like 100) 😅 . At the moment I've changed the default threshold, but it's not a feasible solution if I want to keep up to date with the commits in the YOLOv5 project.

Let's assume that we do end up increasing ar_thr to say 100. But there could always be use cases that have objects with even higher aspect ratios - especially for those working with custom datasets.

If you have thoughts on any other way to give users this flexibility (and yet not have any side-effects on existing datasets or congest the code base), I'd be willing to work on a PR.

[MrinalJain17]

@MrinalJain17 yes option 2 sounds pretty cool.

But wait, maybe this is not a good idea. In this workflow, in a dataset with many low AR objects, i.e. VOC, where things are mostly square, our augmented mosaic loader will crop these annotations randomly, and this candidate criteria is in place as a threshold to help reject unwanted crops. i.e. if a vehicle is 90% cropped in one direction it's AR will multiply by 10, exceeding the un-augmented dataset AR stats, but we may still want to retain this cropped vehicle for training.

It's not clear what the best route is. VOC is easily trainable, we could run a study training it at various ar_thr and simply increase the default threshold.

@glenn-jocher How about this - We add a flag (like auto_ar_thr) to the training script. And perform the auto solve operation only if that flag is enabled. Otherwise use the default value of 20.

[glenn-jocher]

@MrinalJain17 maybe we could solve for dataset_max_ar, and then ar_thr = max(dataset_max_ar, 20). This would satisfy your ar 100 dataset, while also retaining a high 20 ar when square objects get cropped during augmentation in datasets like VOC.

Or maybe we should just drop AR thresholds (or increase beyond the point of meaningfulness) for training data. The main purpose there is to avoid errors in labels (i.e. zero-width labels) rather than to filter correct labels, but also to align the labels with what the model can actually predict, since YOLO models are not set up to detect very high AR objects with their default convolution kernals.

YOLO relies on square 1x1 and 3x3 convolutions. If you have these sorts of objects you may be better suited with alternative kernel structures, i.e. 1x5 and 5x1, or longer kernels perhaps in place of, or in addition to the existing 3x3 kernels. You'd have to experiment to see what works best there.

[MrinalJain17]

@glenn-jocher Thank you for the clarification. I definitely have a better understanding of why ar_thr is there in the first place.

I also liked your idea to compute a dataset_max_ar, and I've updated the PR accordingly. Let me know what you think.

[MrinalJain17]

@glenn-jocher Any suggestions/recommendations for this?

[glenn-jocher]

@MrinalJain17 thanks for the updates! The dataset calculation line is succinct, nice work.

I'm going to run a simple experiment though, since it's been a long time since we evaluated the effect of this parameter. Hopefully the simplest solution is just to raise this threshold. Results should log to https://wandb.ai/glenn-jocher/test_VOC_ar_thr

# VOC
import os
from itertools import repeat

for b, m, ar in zip(repeat(64), repeat('yolov5m6'), [2, 4, 8, 16, 32, 64, 128, 256]):  # zip(batch, model, ar_thr)
  os. environ['AR_THR'] = str(ar)
  !python train.py --batch {b} --weights {m}.pt --data VOC.yaml --epochs 50 --cache --img 512 --nosave --hyp hyp.finetune.yaml --project test_VOC_ar_thr --name {m}-{ar}

with

import os 

def box_candidates(box1, box2, wh_thr=2, ar_thr=20, area_thr=0.1, eps=1e-16):  # box1(4,n), box2(4,n)
    # Compute candidate boxes: box1 before augment, box2 after augment, wh_thr (pixels), aspect_ratio_thr, area_ratio
    ar_thr = int(os.getenv('AR_THR'))
    w1, h1 = box1[2] - box1[0], box1[3] - box1[1]
    w2, h2 = box2[2] - box2[0], box2[3] - box2[1]
    ar = np.maximum(w2 / (h2 + eps), h2 / (w2 + eps))  # aspect ratio
    return (w2 > wh_thr) & (h2 > wh_thr) & (w2 * h2 / (w1 * h1 + eps) > area_thr) & (ar < ar_thr)  # candidates

[glenn-jocher]

@MrinalJain17 AR study on VOC is finished. Results shown W&B here:
https://wandb.ai/glenn-jocher/test_VOC_ar_thr?workspace=user-glenn-jocher

Results mainly show that very low AR thresholds like 4, 8 perform poorly, as expected. Thresholds above these values perform similarly.

[MrinalJain17]

@glenn-jocher Great to see the results. Thanks for conducting those experiments. Really appreciate your efforts. 😄

Hope to have fix merged soon. Rebased to master

[glenn-jocher]

@MrinalJain17 found a small bug in PR. The AR computation is in normalized-space here, which will only provide a correct result if the image is square. The proper way is to compute AR in pixel-space (multiply w and h by image width and image height prior to computing AR).

[MrinalJain17]

Ooh, that's a sneaky bug. Nice catch!

I've updated the PR to fix it. @glenn-jocher

Rebased to master

[MrinalJain17]

@glenn-jocher Any new updates on this?

PS: Apologies for nagging 😅

[MrinalJain17]

@glenn-jocher Seems like this PR can be merged. Any other changes that you would recommend?

EDIT: Rebased to master.

[MrinalJain17]

Hey @glenn-jocher Do you have any other suggestions/thoughts for this PR? I know the ar computation is not as compact after the fix for non-square images, but it seems stable now.

[MrinalJain17]

Hi @glenn-jocher

Still waiting for your approval on this PR. Or perhaps any changes?

[MrinalJain17]

Hey @glenn-jocher I was wondering if this PR is still under consideration. I can close it out otherwise. Hopefully there be some other fix to make Yolov5 work with slender objects better.

[glenn-jocher]

@MrinalJain17 I think probably based on the earlier study we might simply want to increase the base AR threshold, from 20 probably to 100, which should cover almost all the use cases I can think of.

We're trying to strike the right compromise between complication and capability, and I'll be the first to admit the train/val loader is already quite complicated.

What do you think?

[glenn-jocher]

@MrinalJain17 also I don't know if we want to set the threshold based on edge cases, i.e. if there are a few bad outliers in the data that show extreme ARs we may not want to use them as the basis for deciding the threshold.

Typically on a noisy (gaussian or otherwise) distribution you might clip the tails at say 3-sigma or 6-sigma, but of course this just adds additional complications and an additional parameter.

[MrinalJain17]

@MrinalJain17 I think probably based on the earlier study we might simply want to increase the base AR threshold, from 20 probably to 100, which should cover almost all the use cases I can think of.

We're trying to strike the right compromise between complication and capability, and I'll be the first to admit the train/val loader is already quite complicated.

What do you think?

@glenn-jocher Agreed, based on the experiment you conducted earlier, it seems like increasing the threshold would be the most straightforward (and safe) way.

And like you mentioned, using the mean + 3/4 SD could be a more robust solution, but probably not worth the added complexity.

I can update the PR and set ar_thr to 100.

[glenn-jocher]

@MrinalJain17 PR is merged. Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐

Thanks for all the effort on this! This will surely help future users get better results on a wider variety of datasets.