Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

min max wrapper #556

Merged
merged 86 commits into from
Nov 17, 2021
Merged

Conversation

janhenriklambrechts
Copy link
Contributor

@janhenriklambrechts janhenriklambrechts commented Sep 30, 2021

Before submitting

  • Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure to update the docs?
  • Did you write any new necessary tests?

What does this PR do?

Fixes #49

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

@Borda Borda marked this pull request as draft October 1, 2021 06:56
@Borda Borda changed the title WIP / Test49/min max metric min max metric Oct 1, 2021
@Borda
Copy link
Member

Borda commented Oct 1, 2021

@SkafteNicki seems to be a duplicate to #506

@SkafteNicki SkafteNicki mentioned this pull request Oct 1, 2021
4 tasks
torchmetrics/wrappers/minmax.py Outdated Show resolved Hide resolved
torchmetrics/wrappers/minmax.py Outdated Show resolved Hide resolved
torchmetrics/wrappers/minmax.py Outdated Show resolved Hide resolved
torchmetrics/wrappers/minmax.py Outdated Show resolved Hide resolved
torchmetrics/wrappers/__init__.py Outdated Show resolved Hide resolved
@Borda
Copy link
Member

Borda commented Oct 13, 2021

@janhenriklambrechts how is it going here, ready for review?

@janhenriklambrechts
Copy link
Contributor Author

@Borda not yet finished, but will work on it this afternoon -- so expect a clean PR by tomorrow :)

@codecov
Copy link

codecov bot commented Oct 13, 2021

Codecov Report

Merging #556 (297943b) into master (93cb842) will decrease coverage by 0%.
The diff coverage is 94%.

@@          Coverage Diff          @@
##           master   #556   +/-   ##
=====================================
- Coverage      95%    95%   -0%     
=====================================
  Files         152    153    +1     
  Lines        5362   5396   +34     
=====================================
+ Hits         5102   5134   +32     
- Misses        260    262    +2     

@janhenriklambrechts
Copy link
Contributor Author

@Borda @SkafteNicki, I have identified the difference between tests/wrappers/test_minmax.py::compare_fn() and tests/helpers/testers.py::_class_test() that makes the tests fail. However, my understanding of ddp is quite limited so I need some help to understand what the correct behaviour is:

For, compare_fn(), the base_metric gets correctly calculated per batch.
For our test over 10 batches, the raw values of the metric look like this:

0.375
0.34375
0.2916666567325592
0.265625
0.2750000059604645
0.2604166567325592
0.25
0.24609375
0.2465277761220932
0.24375000596046448

The corresponding raw; min; max-array looks like this:

[array(0.24375, dtype=float32), array(0.24375, dtype=float32), array(0.375, dtype=float32)]

These results make sense to me.
However, if we run it within the tests/helpers/testers.py::_class_test(), our raw batch_result for WorldSize=2 is:

0.3438
0.2656
0.2604
0.2461
0.2438

These values correspond to the ones above, but omit half and thus lead to wrong min and max values.
Could you point me to the right direction in what causes this disparity? And how I could fix this?

@Borda
Copy link
Member

Borda commented Nov 2, 2021

@SkafteNicki mind have look ^^ 🐰

@mergify mergify bot removed the has conflicts label Nov 15, 2021
@mergify mergify bot added the ready label Nov 15, 2021
@Borda Borda merged commit 1a041bb into Lightning-AI:master Nov 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

MinMaxMetric for wrapping other metrics
4 participants