NeurIPS-2020 Big Self-Supervised Models are Strong Semi-Supervised Learners

2020-12-06 · NeurIPS· Vision · semi-supervised learning

Paper: Big Self-Supervised Models are Strong Semi-Supervised Learners

Code: https://github.com/google-research/simclr

SimCLR v2 and semi-supervised learning

For improvements over SimCLR, the work uses larger models and a deeper projection head; empirically two layers suffice, and it references a momentum encoder.
The semi-supervised pipeline first learns a strong representation via self-supervised contrastive learning, then fine-tunes on a small labeled set. After fine-tuning, that model acts as a teacher to generate pseudo-labels so self-training can use more unlabeled data—following the spirit of Google’s 2019 Noisy Student.

2.1 Noisy Student trains a teacher on ImageNet, generates pseudo-labels, and uses them to train a student model.