CrossCLR: cross-modal contrastive learning for multi-modal video representations

Mohammadreza Zolfaghari, Y. Zhu, P. Gehler, Thomas Brox
IEEE International Conference on Computer Vision (ICCV), 2021
Abstract: Contrastive learning allows us to flexibly define powerful losses by contrasting positive pairs from sets of negative samples. Recently, the principle has also been used to learn cross-modal embeddings for video and text, yet without exploiting its full potential. In particular, previous losses do not take the intra-modality similarities into account, which leads to inefficient embeddings, as the same content is mapped to multiple points in the embedding space. With CrossCLR, we present a contrastive loss that fixes this issue. Moreover, we define sets of highly related samples in terms of their input embeddings and exclude them from the negative samples to avoid issues with false negatives. We show that these principles consistently improve the quality of the learned embeddings. The joint embeddings learned with CrossCLR extend the state of the art in video-text retrieval on Youcook2 and LSMDC by a large margin. We also demonstrate the generality of the concept by learning improved joint embeddings for other pairs of modalities.

Images and movies


BibTex reference

  author       = "M. Zolfaghari and Y. Zhu and P. Gehler and T. Brox",
  title        = "CrossCLR: cross-modal contrastive learning for multi-modal video representations",
  booktitle    = "IEEE International Conference on Computer Vision (ICCV)",
  month        = " ",
  year         = "2021",
  url          = "http://lmb.informatik.uni-freiburg.de/Publications/2021/ZB21"

Other publications in the database