Grounding Calculation

[krbuettner]

❓ Questions and Help

Hi, I am looking at the grounding head and am wondering if you could provide clarification on some items in grounding_head.py... what is the difference between local_similarity and local_distance and why is local_distance used to calculate global_dist_r2w rather than local_similarity? Also, does the sign of the grounding score matter? Are high scores large positive #s and low scores small negative #s?

Thank you for your time and help.

[alirezazareian]

Similarity and distance are opposite concepts here. The reason we have both is to support various metrics. E.g. cosine measures similarity (high means similar) while Euclidean is a distance (high means not similar). For generality, we define both similarity and distance for each metric. Similarity is used to compute attention weights, since higher similarity means more attention. Then attention is used to get a weighted sum of the local distance matrix and determine global distances between images and captions in the batch. The global distance is used to compute the loss, since loss is higher if the distance of the image to its corresponding caption is high.

Distance is always positive except for dot product, since dot product is unbounded in both ways. For dot product, a very negative number for distance means a very low distance.