add diversity evaluation metrics #1416

YanZhangADS · 2021-05-28T17:55:21Z

Description

Related Issues

Checklist:

I have followed the contribution guidelines and code style for this project.
I have added tests covering my contributions.
I have updated the documentation accordingly.
This PR is being made to staging branch and not to main branch.

review-notebook-app · 2021-05-28T17:55:25Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

gramhagen

this looks really great. i haven't had a chance to look closely at the notebook yet though

reco_utils/evaluation/diversity_evaluator.py

tests/unit/reco_utils/evaluation/test_spark_evaluation_diversity_metrics.py

reco_utils/evaluation/diversity_evaluator.py

tests/unit/reco_utils/evaluation/test_spark_evaluation_diversity_metrics.py

…imilarity matrix

gramhagen

this is looking great! i took a closer look at the notebook and focused most of the suggestions there, but caught a couple small items in the main and test files.

examples/03_evaluate/als_movielens_diversity_metrics.ipynb

tests/unit/reco_utils/evaluation/test_spark_diversity_evaluator.py

examples/03_evaluate/als_movielens_diversity_metrics.ipynb

reco_utils/evaluation/spark_diversity_evaluator.py

miguelgfierro

Yan this is really good, I made some suggestions

reco_utils/evaluation/spark_diversity_evaluator.py

tests/unit/reco_utils/evaluation/test_spark_diversity_evaluator.py

miguelgfierro · 2021-06-05T06:28:58Z

tests/unit/reco_utils/evaluation/test_spark_diversity_evaluator.py

+def test_catalog_coverage(evaluator, target_metrics):
+
+    c_coverage = evaluator.catalog_coverage()
+    assert c_coverage == target_metrics["c_coverage"]
+
+@pytest.mark.spark
+def test_distributional_coverage(evaluator, target_metrics):
+
+    d_coverage = evaluator.distributional_coverage()
+    assert d_coverage == target_metrics["d_coverage"]
+
+@pytest.mark.spark
+def test_item_novelty(evaluator, target_metrics):
+    actual = evaluator.item_novelty().toPandas()
+    assert_frame_equal(target_metrics["item_novelty"], actual, check_exact=False, check_less_precise=4)
+
+@pytest.mark.spark    
+def test_user_novelty(evaluator, target_metrics):
+    actual = evaluator.user_novelty().toPandas()
+    assert_frame_equal(target_metrics["user_novelty"], actual, check_exact=False, check_less_precise=4)
+
+@pytest.mark.spark    
+def test_novelty(evaluator, target_metrics):
+    actual = evaluator.novelty().toPandas()
+    assert_frame_equal(target_metrics["novelty"], actual, check_exact=False, check_less_precise=4)


We also need to check the limits, this is very important to check that the formulas are correct. This would be check perfect novelty and non-novelty, perfect diversity and non-diversity, etc.

See example: https://github.com/microsoft/recommenders/blob/main/tests/unit/reco_utils/evaluation/test_spark_evaluation.py#L104

"check the limits" -- what is the precision you would recommend? I am comparing the number to the 4 positions after the decimal point.

4 position is great, we even have 2 position in some cases

tests/unit/reco_utils/evaluation/test_spark_diversity_evaluator.py

examples/03_evaluate/als_movielens_diversity_metrics.ipynb

miguelgfierro · 2021-06-05T06:45:43Z

I think we should add an entry about the diversity metrics in this notebook: https://github.com/microsoft/recommenders/blob/main/examples/03_evaluate/evaluation.ipynb
Basically do the same example as we did with rating and ranking, setting the limits, etc

reco_utils/evaluation/spark_diversity_evaluation.py

gramhagen

Great job!

reco_utils/evaluation/spark_diversity_evaluation.py

add diversity evaluation metrics

7cbc66d

YanZhangADS requested review from anargyri, gramhagen, loomlike, miguelgfierro, wutaomsft and yueguoguo as code owners May 28, 2021 17:55

fix typo

406e8e0

gramhagen requested changes May 28, 2021

View reviewed changes

YanZhangADS added 14 commits June 1, 2021 15:16

formating using black

d28167e

modify licence

f565ff1

replace == None with is None

ec93a79

enhance code

1fe412f

formatting

b968b42

fix

8ae928b

fix

d75370d

optimize logic for calculating serendipity, removing full matrix of s…

c21ce80

…imilarity matrix

add docstring

1204909

change file name

ce954e5

fix import

1e70a71

changed file name

a5631db

fix

42700d5

evaluation example notebook

a2974cd

gramhagen requested changes Jun 4, 2021

View reviewed changes

miguelgfierro requested changes Jun 5, 2021

View reviewed changes

miguelgfierro reviewed Jun 5, 2021

View reviewed changes

examples/03_evaluate/als_movielens_diversity_metrics.ipynb Show resolved Hide resolved

YanZhangADS added 2 commits June 6, 2021 11:23

fix

2ccae9a

fix

72bf0f7

YanZhangADS added 5 commits June 6, 2021 11:26

change file name

42bc562

fix

f4be166

change file name

47d4fba

remove div evaluator fixture

dae9b0f

fix

6dc27aa

gramhagen reviewed Jun 8, 2021

View reviewed changes

reco_utils/evaluation/spark_diversity_evaluation.py Outdated Show resolved Hide resolved

YanZhangADS added 9 commits June 9, 2021 02:28

fix input variable names

f9b14d0

fix typo

c9e85da

improve docstring

8a126e1

fix

f3148e7

improve ALS example notebook

5501d6f

improve ALS example notebook

334e2ba

optimize code

cda1bbc

fix

747faf1

add reference

fa1c4f8

gramhagen approved these changes Jun 10, 2021

View reviewed changes

anargyri reviewed Jun 11, 2021

View reviewed changes

reco_utils/evaluation/spark_diversity_evaluation.py Show resolved Hide resolved

improve docstring

dcb1b1a

miguelgfierro approved these changes Jun 11, 2021

View reviewed changes

miguelgfierro merged commit dd82287 into staging Jun 11, 2021

miguelgfierro deleted the zhangya_diversitymetrics branch June 11, 2021 22:09

anargyri mentioned this pull request Jun 16, 2021

Improvements on diversity metrics #1453

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add diversity evaluation metrics #1416

add diversity evaluation metrics #1416

YanZhangADS commented May 28, 2021 •

edited by anargyri

Loading

review-notebook-app bot commented May 28, 2021

gramhagen left a comment

gramhagen left a comment

miguelgfierro left a comment

miguelgfierro Jun 5, 2021

YanZhangADS Jun 7, 2021 •

edited

Loading

miguelgfierro Jun 9, 2021

miguelgfierro commented Jun 5, 2021

gramhagen left a comment

add diversity evaluation metrics #1416

add diversity evaluation metrics #1416

Conversation

YanZhangADS commented May 28, 2021 • edited by anargyri Loading

Description

Related Issues

Checklist:

review-notebook-app bot commented May 28, 2021

gramhagen left a comment

Choose a reason for hiding this comment

gramhagen left a comment

Choose a reason for hiding this comment

miguelgfierro left a comment

Choose a reason for hiding this comment

miguelgfierro Jun 5, 2021

Choose a reason for hiding this comment

YanZhangADS Jun 7, 2021 • edited Loading

Choose a reason for hiding this comment

miguelgfierro Jun 9, 2021

Choose a reason for hiding this comment

miguelgfierro commented Jun 5, 2021

gramhagen left a comment

Choose a reason for hiding this comment

YanZhangADS commented May 28, 2021 •

edited by anargyri

Loading

YanZhangADS Jun 7, 2021 •

edited

Loading