NeurIPS-2020 Bootstrap your own latent：A new approach to self-supervised Learning

2020-12-06 · NeurIPS· Vision · self-supervised learning

Paper: Bootstrap your own latent: A new approach to self-supervised Learning

BYOL: No negative samples—turn matching into a prediction task

In contrastive learning, negative samples provide a necessary constraint. Optimizing with positive pairs alone makes shortcut solutions easy: when similar instances are pushed toward similar features, the model can map every input to the same output so all features collapse, the loss goes to zero, and nothing is learned (representation collapse). Negatives require dissimilar instances to stay apart as well as similar ones to align; identical outputs for everything would incur large loss on negatives and force more discriminative features.
Forward pass: $f_\theta$ is updated by gradients; $f_\xi$ is a momentum encoder; $g_\theta$ and $q_\theta$ are MLP layers. The upper branch acts as the query encoder and the lower as the key encoder. The two branches use different targets and are trained with MSE loss.

avatar