DeepMind-2018 Representation Learning with Contrastive Predictive Coding

· DeepMind· · vision-language, contrastive-learning

Paper: Representation Learning with Contrastive Predictive Coding

CPC: Predictive proxy tasks; models generalize to text, audio, images, and more

  1. An encoder maps inputs up to time t to representations z; an autoregressive model (RNN/LSTM, etc.) aggregates them into a context c_t that is sufficiently informative to support prediction (denoted pre).
  2. Positive samples are the encoded representations z from inputs after time t; pre acts as the query and the future z plays the role of the positive sample, enabling contrastive learning.

<HR align=left color=#987cb9 SIZE=1>