CVPR-2023 Image as a Foreign Language：BEiT Pretraining for All Vision and Vision-Language Tasks

2023-01-15 · CVPR· Vision · vision-language

Paper: Image as a Foreign Language：BEiT Pretraining for All Vision and Vision-Language Tasks

Code: https://aka.ms/beit-3

BEIT-3: Multimodal Masked Language Model

Abstract

The work introduces multi-way transformers and applies masked language modeling to images (“Imglish”), text (English), and image–text pairs treated as parallel sentences.

Introduction: The Big Convergence

Language, vision, and multimodal learning are moving toward a unified foundation-model paradigm; this paper aims to advance that unified-framework line further. The convergence is driven mainly by three trends:

Transformers have spread from NLP into CV and multimodal modeling. For vision–language tasks, dual-encoder models suit fast retrieval, encoder–decoder stacks suit generation, and fusion-encoder designs are strong for joint image–text encoding. Even so, most foundation models still need architectural or interface tweaks to match downstream input–output formats.
Masked modeling transfers well across modalities. Stacking many auxiliary losses hurts efficiency, and balancing their weights is hard—some objectives complement each other while others compete, which makes tuning largely manual. The authors therefore rely on a single MLM objective: every modality is a sequence of tokens, and an image paired with text can be read as parallel sentences (sentence 1 followed by sentence 2).
Scaling model and data capacity: only a sufficiently large model can cover diverse tasks, so parameters are pushed to the billion scale while pretraining still draws on public data resources.

BEIT-3: A General-Purpose Multimodal Foundation Model

Architecturally the model follows VLMO: early multi-head self-attention is shared across modalities, while the feed-forward networks (FFNs) are modality-specific. Pretraining masks either the image or the text—both are cloze-style prediction tasks.