MM-2022 Can Language Understand Depth？

· MM· · vision-language, contrastive-learning

Paper: Can Language Understand Depth?

Code: https://github.com/Adonis-galaxy/DepthCLIP

DepthCLIP: reframing depth estimation from regression to classification with CLIP

Abstract

Transferring semantic knowledge learned from CLIP to more complex quantitative target tasks—such as depth estimation that relies on geometric cues—remains difficult. This work applies CLIP to zero-shot depth estimation, termed DepthCLIP.

Method

avatar

Depth estimation is cast from regression to classification: depth is partitioned into coarse bins, described with the near/far language prompts in Figure 2 and tied to numeric values; similarity is obtained from the product of text and visual features.

Experiment

avatar
avatar

<HR align=left color=#987cb9 SIZE=1>