Add AverageMeter implementation #138

alanhdu · 2021-03-26T20:57:29Z

This adds an AverageMeter, which is a simple metric that takes the average of a stream of values.

Closes #129

pep8speaks · 2021-03-26T20:57:34Z

Hello @alanhdu! Thanks for updating this PR.

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-04-14 18:29:03 UTC

codecov · 2021-03-26T20:58:38Z

Codecov Report

Merging #138 (4bfc246) into master (ec239e0) will increase coverage by 0.13%.
The diff coverage is 96.90%.

❗ Current head 4bfc246 differs from pull request most recent head c205ffd. Consider uploading reports for the commit c205ffd to get more accurate results

@@            Coverage Diff             @@
##           master     #138      +/-   ##
==========================================
+ Coverage   95.99%   96.12%   +0.13%     
==========================================
  Files         168       90      -78     
  Lines        5144     2790    -2354     
==========================================
- Hits         4938     2682    -2256     
+ Misses        206      108      -98

Flag	Coverage Δ
Linux	`79.78% <91.85%> (+1.05%)`	⬆️
Windows	`79.78% <91.85%> (+1.05%)`	⬆️
cpu	`96.12% <96.90%> (+0.13%)`	⬆️
gpu	`?`
macOS	`96.12% <96.90%> (+0.13%)`	⬆️
pytest	`96.12% <96.90%> (+0.13%)`	⬆️
python3.6	`?`
python3.8	`96.12% <96.90%> (+0.13%)`	⬆️
python3.9	`96.02% <95.57%> (+0.02%)`	⬆️
torch1.3.1	`?`
torch1.4.0	`95.16% <96.46%> (+0.17%)`	⬆️
torch1.8.1	`96.02% <95.57%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
torchmetrics/metric.py	`94.75% <66.66%> (-0.32%)`	⬇️
torchmetrics/utilities/imports.py	`82.14% <75.00%> (+2.14%)`	⬆️
torchmetrics/functional/regression/spearman.py	`93.02% <93.02%> (ø)`
torchmetrics/average.py	`95.83% <95.83%> (ø)`
torchmetrics/functional/regression/pearson.py	`96.00% <96.00%> (ø)`
torchmetrics/__about__.py	`100.00% <100.00%> (ø)`
torchmetrics/__init__.py	`100.00% <100.00%> (ø)`
torchmetrics/classification/__init__.py	`100.00% <100.00%> (ø)`
torchmetrics/classification/accuracy.py	`100.00% <100.00%> (ø)`
...hmetrics/classification/binned_precision_recall.py	`100.00% <100.00%> (ø)`
... and 100 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ec239e0...c205ffd. Read the comment docs.

alanhdu · 2021-03-26T20:59:04Z

I am currently struggling a bit with the test suite -- many of them seem to have hardcoded an interface where metrics operate over predictions and target, which isn't applicable here. I will continue to study the tests and see how to add it.

I was also not sure of the code organization -- I decided to just create a new submodule since it doesn't seem to fit any of the existing module's super well, but I'm totally open to guidance on where this should live.

Borda · 2021-03-28T11:38:17Z

I am currently struggling a bit with the test suite

ping us in slack if you need our help :]

SkafteNicki

For testing, I think it should be quite easily to check that this works as intended by using our current testing interface and just compare against np.average.

Also missing:

add instance to changelog
add ref in the docs/source/references/modules.rst

torchmetrics/average.py

alanhdu · 2021-03-31T21:09:22Z

Update: I'm quite busy at work this week with other things, but I will try to take a look at this next week.

alanhdu · 2021-04-05T19:40:30Z

Find some time to come back to this. I made a couple different changes:

I believe I responded to all the style requests
I changed the interface slightly to allow AverageMeter to take in Tensor values/weights along with scalar ones. If it has a tensor value, then we treat each item of the Tensor as a separate observation. Weights is automatically broadcasted to fit the values
I have added tests, although I had to abuse some of the name slightly.

tests/test_average.py

torchmetrics/average.py

alanhdu · 2021-04-05T20:10:21Z

Intersting... it looks like torch.broadcast_to was only added in PyTorch 1.8. Should I add a version check here, or just revert that behavior (and require weights to either be a scalar or the same shape as values)?

torchmetrics/average.py

tests/test_average.py

alanhdu · 2021-04-12T19:24:24Z

I believe I've responded to the code review comments. The tests for the scalar case are a little messy, but I think they should be workable.

SkafteNicki

PR is looking good, if you could look into failing test that would be great :]

It seems something is wrong with the type checking:
Cannot resolve forward reference in type annotations of "torchmetrics.AverageMeter.update": name 'Union' is not defined

CHANGELOG.md

Borda · 2021-04-13T14:29:56Z

@alanhdu @SkafteNicki how are we going here? 🐰

Borda · 2021-04-14T15:38:54Z

torchmetrics/average.py

+        value: "typing.Union[Tensor, float]",
+        weight: "typing.Union[Tensor, float]" = 1.0


not very happy about this typing, can we get some better way?

it is changed to just Union[Tensor, float] now, is that better?
else we can just go with Any (which is not completely correct, but should not result in the pickle problem)

torchmetrics/average.py

* init files * rest * pep8 * changelog * clamp * suggestions * rename * format * _sk_pearsonr * inline * fix sync * fix tests * fix docs * Apply suggestions from code review * Update torchmetrics/functional/regression/pearson.py * atol * update * pep8 * pep8 * chlog * . Co-authored-by: Jirka Borovec <[email protected]> Co-authored-by: Jirka Borovec <[email protected]>

* ranking * init files * update * nearly working * fix tests * pep8 * add docs * fix doctests * fix docs * pep8 * isort * ghlog * Apply suggestions from code review Co-authored-by: Jirka Borovec <[email protected]>

* added test changes * fix style error * fixed typo * added changes for requires_grad * metrics differentiability testing generalization * Update tests/classification/test_accuracy.py Co-authored-by: Nicki Skafte <[email protected]> * fix tests * pep8 * changelog * fix docs * fix tests * pep8 * Apply suggestions from code review Co-authored-by: Nicki Skafte <[email protected]> Co-authored-by: Jirka Borovec <[email protected]>

* WIP: Binned PR-related metrics * attempt to fix types * switch to linspace to make old pytorch happy * make flake happy * clean up * Add more testing, move test input generation to the approproate place * bugfixes and more stable and thorough tests * flake8 * Reuse python zip-based implementation as it can't be reproduced with torch.where/max * address comments * isort * Add docs and doctests, make APIs same as non-binned versions * pep8 * isort * doctests likes longer title underlines :O * use numpy's nan_to_num * add atol to bleu tests to make them more stable * atol=1e-2 for bleu * add more docs * pep8 * remove nlp test hack * address comments * pep8 * abc * flake8 * remove typecheck * chlog Co-authored-by: Jirka Borovec <[email protected]> Co-authored-by: Nicki Skafte <[email protected]> Co-authored-by: Jirka Borovec <[email protected]>

* version + about * flake8 * try * .

SkafteNicki added the enhancement New feature or request label Mar 28, 2021

SkafteNicki reviewed Mar 28, 2021

View reviewed changes

alanhdu added 2 commits April 5, 2021 15:32

Add AverageMeter

ea920bf

Fix type annotation to accomodate Python 3.6 bug

854e759

alanhdu force-pushed the average branch from d56f371 to 132775c Compare April 5, 2021 19:36

alanhdu added 3 commits April 5, 2021 15:36

Add tests

69e15a1

Update changelog

1c773d6

Add AverageMeter to docs

0173b98

alanhdu force-pushed the average branch from 132775c to 0173b98 Compare April 5, 2021 19:36

fixup! Add AverageMeter to docs

6a51ce1

alanhdu commented Apr 5, 2021

View reviewed changes

tests/test_average.py Outdated Show resolved Hide resolved

torchmetrics/average.py Outdated Show resolved Hide resolved

maximsch2 reviewed Apr 9, 2021

View reviewed changes

torchmetrics/average.py Outdated Show resolved Hide resolved

torchmetrics/average.py Outdated Show resolved Hide resolved

tests/test_average.py Outdated Show resolved Hide resolved

alanhdu added 3 commits April 12, 2021 12:57

Code review comments

462dc0b

Add tests for scalar case

05bc802

Merge branch 'master' into average

c92e135

alanhdu added 2 commits April 12, 2021 15:28

Fix behavior on PyTorch <1.8

cd883e8

fixup! Add tests for scalar case

ce12f5e

alanhdu force-pushed the average branch from b857237 to ce12f5e Compare April 12, 2021 19:28

fixup! fixup! Add tests for scalar case

4edeb74

alanhdu marked this pull request as ready for review April 12, 2021 20:12

alanhdu requested review from ananyahjha93 and Borda as code owners April 12, 2021 20:12

alanhdu requested review from edenlightning and tchaton as code owners April 12, 2021 20:12

SkafteNicki reviewed Apr 13, 2021

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

Update CHANGELOG.md

e213bc5

Borda requested review from justusschock and SkafteNicki April 13, 2021 14:29

SkafteNicki approved these changes Apr 14, 2021

View reviewed changes

SkafteNicki added the ready label Apr 14, 2021

Borda enabled auto-merge (squash) April 14, 2021 12:50

Borda reviewed Apr 14, 2021

View reviewed changes

auto-merge was automatically disabled April 14, 2021 16:14
Head branch was pushed to by a user without write access

Borda reviewed Apr 14, 2021

View reviewed changes

torchmetrics/average.py Show resolved Hide resolved

SkafteNicki and others added 9 commits April 14, 2021 20:28

Spearman correlation coefficient (Lightning-AI#158)

06c13fd

* ranking * init files * update * nearly working * fix tests * pep8 * add docs * fix doctests * fix docs * pep8 * isort * ghlog * Apply suggestions from code review Co-authored-by: Jirka Borovec <[email protected]>

version + about (Lightning-AI#170)

f5e8dfa

* version + about * flake8 * try * .

fix doc

59ca4da

overload sig

36bde32

fix

fa2ea27

Different import style

c205ffd

Borda force-pushed the average branch from 4bfc246 to c205ffd Compare April 14, 2021 18:28

Borda approved these changes Apr 14, 2021

View reviewed changes

Borda enabled auto-merge (squash) April 14, 2021 18:29

Borda merged commit da00174 into Lightning-AI:master Apr 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add AverageMeter implementation #138

Add AverageMeter implementation #138

alanhdu commented Mar 26, 2021

pep8speaks commented Mar 26, 2021 •

edited

Loading

codecov bot commented Mar 26, 2021 •

edited

Loading

alanhdu commented Mar 26, 2021 •

edited

Loading

Borda commented Mar 28, 2021

SkafteNicki left a comment

alanhdu commented Mar 31, 2021

alanhdu commented Apr 5, 2021

alanhdu commented Apr 5, 2021

alanhdu commented Apr 12, 2021

SkafteNicki left a comment

Borda commented Apr 13, 2021

Borda Apr 14, 2021 •

edited

Loading

SkafteNicki Apr 14, 2021

		value: "typing.Union[Tensor, float]",
		weight: "typing.Union[Tensor, float]" = 1.0

Add AverageMeter implementation #138

Add AverageMeter implementation #138

Conversation

alanhdu commented Mar 26, 2021

pep8speaks commented Mar 26, 2021 • edited Loading

Comment last updated at 2021-04-14 18:29:03 UTC

codecov bot commented Mar 26, 2021 • edited Loading

Codecov Report

alanhdu commented Mar 26, 2021 • edited Loading

Borda commented Mar 28, 2021

SkafteNicki left a comment

Choose a reason for hiding this comment

alanhdu commented Mar 31, 2021

alanhdu commented Apr 5, 2021

alanhdu commented Apr 5, 2021

alanhdu commented Apr 12, 2021

SkafteNicki left a comment

Choose a reason for hiding this comment

Borda commented Apr 13, 2021

Borda Apr 14, 2021 • edited Loading

Choose a reason for hiding this comment

SkafteNicki Apr 14, 2021

Choose a reason for hiding this comment

pep8speaks commented Mar 26, 2021 •

edited

Loading

codecov bot commented Mar 26, 2021 •

edited

Loading

alanhdu commented Mar 26, 2021 •

edited

Loading

Borda Apr 14, 2021 •

edited

Loading