Support for segmentation extreme batch cases #1282

shaydeci · 2023-07-16T11:41:50Z

Added a base class, for visualizing the worst/best batches according to some metric, loss or loss component.
Added segmentation support.
Added some missing pieces that were needed in phase context.
Full DDP support- aggregation of metrics, the best/worst predictions, and averaging the loss.

Note: images can be resized individually inside the logger.

src/super_gradients/training/utils/callbacks/callbacks.py

Louis-Dupont

LGTM overall; but I think the issue I raised with the metrics is quiet important. I know we currently don't have a clean way to handle the metrics but I think we should honestly think about it

src/super_gradients/training/utils/callbacks/callbacks.py

…cb' into feature/SG-901_worst_samples_od_cb

src/super_gradients/training/utils/callbacks/callbacks.py

…cb' into feature/SG-901_worst_samples_od_cb

…t_samples_od_cb

src/super_gradients/training/metrics/segmentation_metrics.py

BloodAxe

Two important changes that I think this PR is missing:

Please add a unit tests that verity this callback works as intended.
Also, adding a docs to show we have this feature would help to increase it's exposure.

src/super_gradients/training/utils/distributed_training_utils.py

…cb' into feature/SG-901_worst_samples_od_cb

src/super_gradients/training/utils/distributed_training_utils.py

src/super_gradients/training/utils/callbacks/callbacks.py

…cb' into feature/SG-901_worst_samples_od_cb

…t_samples_od_cb # Conflicts: # tests/deci_core_unit_test_suite_runner.py

BloodAxe

LGTM

shaydeci and others added 2 commits July 16, 2023 14:37

tested version

5c45ee3

Merge branch 'master' into feature/SG-901_worst_samples_od_cb

14008cf

shaydeci marked this pull request as ready for review July 16, 2023 15:03

shaydeci requested review from ofrimasad, BloodAxe and Louis-Dupont as code owners July 16, 2023 15:03

Louis-Dupont reviewed Jul 17, 2023

View reviewed changes

src/super_gradients/training/utils/callbacks/callbacks.py Outdated Show resolved Hide resolved

Louis-Dupont reviewed Jul 17, 2023

View reviewed changes

src/super_gradients/training/utils/callbacks/callbacks.py Outdated Show resolved Hide resolved

src/super_gradients/training/utils/callbacks/callbacks.py Outdated Show resolved Hide resolved

shaydeci and others added 3 commits July 17, 2023 14:24

changed base to abc and abstractmethod

9bb4c9b

Merge remote-tracking branch 'origin/feature/SG-901_worst_samples_od_…

392c52f

…cb' into feature/SG-901_worst_samples_od_cb

Merge branch 'master' into feature/SG-901_worst_samples_od_cb

f22ec75

BloodAxe reviewed Jul 17, 2023

View reviewed changes

src/super_gradients/training/utils/callbacks/callbacks.py Outdated Show resolved Hide resolved

BloodAxe reviewed Jul 17, 2023

View reviewed changes

src/super_gradients/training/utils/callbacks/callbacks.py Outdated Show resolved Hide resolved

BloodAxe reviewed Jul 17, 2023

View reviewed changes

src/super_gradients/training/utils/callbacks/callbacks.py Outdated Show resolved Hide resolved

shaydeci and others added 5 commits July 17, 2023 20:10

comments wip

bbd2f79

Merge remote-tracking branch 'origin/feature/SG-901_worst_samples_od_…

221cba8

…cb' into feature/SG-901_worst_samples_od_cb

Merge remote-tracking branch 'origin/master' into feature/SG-901_wors…

30d86aa

…t_samples_od_cb

refactoring, docs

6344f83

Merge branch 'master' into feature/SG-901_worst_samples_od_cb

6db796d

shaydeci requested a review from BloodAxe July 24, 2023 06:47

BloodAxe reviewed Jul 24, 2023

View reviewed changes

src/super_gradients/training/metrics/segmentation_metrics.py Outdated Show resolved Hide resolved

BloodAxe suggested changes Jul 24, 2023

View reviewed changes

BloodAxe reviewed Jul 24, 2023

View reviewed changes

src/super_gradients/training/utils/distributed_training_utils.py Outdated Show resolved Hide resolved

shaydeci added 2 commits July 24, 2023 10:43

removed testing metric

2854f35

Merge remote-tracking branch 'origin/feature/SG-901_worst_samples_od_…

7b06812

…cb' into feature/SG-901_worst_samples_od_cb

BloodAxe reviewed Jul 24, 2023

View reviewed changes

src/super_gradients/training/utils/distributed_training_utils.py Show resolved Hide resolved

removed device arg from maybe all reduce

2b8674a

BloodAxe reviewed Jul 24, 2023

View reviewed changes

src/super_gradients/training/utils/callbacks/callbacks.py Outdated Show resolved Hide resolved

BloodAxe reviewed Jul 24, 2023

View reviewed changes

src/super_gradients/training/utils/callbacks/callbacks.py Outdated Show resolved Hide resolved

BloodAxe reviewed Jul 24, 2023

View reviewed changes

src/super_gradients/training/utils/callbacks/callbacks.py Outdated Show resolved Hide resolved

BloodAxe reviewed Jul 24, 2023

View reviewed changes

src/super_gradients/training/utils/callbacks/callbacks.py Outdated Show resolved Hide resolved

shaydeci and others added 6 commits July 24, 2023 11:16

fixed metrice typo in docs

0633b55

loss_name changed to loss_to_monitor

b40599b

Merge branch 'master' into feature/SG-901_worst_samples_od_cb

3671565

unit tests added

d0730f1

Merge remote-tracking branch 'origin/feature/SG-901_worst_samples_od_…

39b36ac

…cb' into feature/SG-901_worst_samples_od_cb

Merge remote-tracking branch 'origin/master' into feature/SG-901_wors…

74827eb

…t_samples_od_cb # Conflicts: # tests/deci_core_unit_test_suite_runner.py

BloodAxe approved these changes Aug 1, 2023

View reviewed changes

shaydeci added 2 commits August 1, 2023 15:31

Merge branch 'master' into feature/SG-901_worst_samples_od_cb

432a184

Merge branch 'master' into feature/SG-901_worst_samples_od_cb

2b17f02

shaydeci merged commit 699b972 into master Aug 1, 2023

shaydeci deleted the feature/SG-901_worst_samples_od_cb branch August 1, 2023 14:04

BloodAxe mentioned this pull request Aug 10, 2023

Segmentation result display #985

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for segmentation extreme batch cases #1282

Support for segmentation extreme batch cases #1282

shaydeci commented Jul 16, 2023 •

edited

Loading

Louis-Dupont left a comment

BloodAxe left a comment •

edited

Loading

BloodAxe left a comment

Support for segmentation extreme batch cases #1282

Support for segmentation extreme batch cases #1282

Conversation

shaydeci commented Jul 16, 2023 • edited Loading

Louis-Dupont left a comment

Choose a reason for hiding this comment

BloodAxe left a comment • edited Loading

Choose a reason for hiding this comment

BloodAxe left a comment

Choose a reason for hiding this comment

shaydeci commented Jul 16, 2023 •

edited

Loading

BloodAxe left a comment •

edited

Loading