NeurIPS-2022 CoCa：Contrastive Captioners are Image-Text Foundation Models

Paper: CoCa：Contrastive Captioners are Image-Text Foundation Models

Code: https://github.com/lucidrains/CoCa-pytorch (reimplementation)

CoCa: a text-side decoder with large-scale pretraining data

Abstract

This work introduces the contrastive captioner (CoCa), trained with both contrastive loss and captioning loss.

Approach

The left branch is an image encoder; the right branch splits the text decoder into two parts. Captioning loss corresponds to language modeling loss. The text decoder on the right is causal from the outset, so a single forward pass suffices, which lowers compute cost.

Pretraining uses a very large corpus—on the order of billions of image–text pairs, well beyond prior scale—which contributes to strong results.

Natural Language Supervision

Dual-Encoder Contrastive Learning

Encoder-Decoder Captioning

Contrastive Captioners Pretraining

Experiments

Each vertex of the polygon corresponds to a task or dataset; unimodal performance is also very strong.

<HR align=left color=#987cb9 SIZE=1>