InfoXLM

The paper written by Chi et al. (2020) introduces theoretical framework which unifies common approaches to pretraining, such as MMLM, cross-lingual models under single view. Furthermore, the paper introduces new pretraining objective based on contrastive learning, which motivates the model to embed translations of the same sentence similarly, while distinguishing them from negative samples.