Beyond-accuracy metric suggestions

[deklanw]

I see in the code there is a todo for some of these metrics. Just going to be putting some notes here about this topic

RecBole/recbole/evaluator/metrics.py

Lines 324 to 337 in 4c4838b

	# Item based Metrics #
	# TODO
	# def coverage_():
	# raise NotImplementedError
	# def gini_index_():
	# raise NotImplementedError
	# def shannon_entropy_():
	# raise NotImplementedError
	# def diversity_():
	# raise NotImplementedError

Noting that many of these metrics depend on some definition of "genre" see #667

It's a bit confusing since the literature seems to use overlapping terms for some of these (all of these able to be defined @ k):

• Number of unique items correctly recommended divided by total number of items (disagreement over whether "correctly" is required?)
  • Coverage, Catalog coverage, Aggregated diversity
• Mean distance (can be defined many ways: Hamming distance of genre, some similarity measure of interaction vectors, etc) between recommended item list for user, averaged over users
  • Diversity, Individual diversity, Intra-list Distance
• Number of unique genres of correctly recommended items for user divided by number of unique genre user interacts with, averaged over users
  • Category coverage, Genre coverage

There are some which seem to lack a unified definition, e.g. novelty: Let the (naive) probability of interacting with an item be the number of users that have interacted with an item divided by the number of users. Then, the novelty of a recommendation list is the average surprise (negative log probability, i.e. self-information) of the recommendation list, averaged of the users.

Not bothering with citations since these are spread around all over.