CVPR-2022 Grounded Language-Image Pre-trainin
GLIP: detection + grounding
Abstract
Object detection and phrase grounding are combined for pre-training. The benefits are: (1) learning from both detection and grounding data to improve the task; (2) exploiting large-scale image–text pairs by generating grounding boxes through self-training, so the learned representations are semantically rich. On COCO and LVIS, zero-shot performance without seeing images during training reaches very high accuracy.
Introduction
At inference, given category labels that are turned into a sentence, GLIP detects the corresponding objects from that sentence—or, as in grounding, one can provide a sentence directly.
Grounded Language Image Pre-training
Unlike CLIP’s two-branch similarity computation, GLIP performs fusion in between.
Unified Formulation
object detection
An image backbone produces region embeddings, followed by a classification head $W$; NMS filters bounding boxes before the loss is computed.
detection as phrase grounding
Matching scores encode how image regions align with words in the sentence.
Similarity between text embeddings and image embeddings yields the final region–word alignment scores $S_{ground}$.
Equivalence between detection and grounding
With small modifications, the two views are unified: the second dimension of $T\in {0,1}^{N\times c}$ in Eq. 2 is extended to match that of $S_{ground}\in R^{N\times M}$ in Eq. 3, because the number of tokens $M$ is always greater than the number of categories $c$; positive and negative matches are then computed.
Experiment
Pre-training datasets differ across methods, so comparisons are not strictly fair, but GLIP clearly performs very well.
<HR align=left color=#987cb9 SIZE=1>