ICCV-2021 Swin Transformer：Hierarchical Vision Transformer using Shifted Windows

2021-08-17 · ICCV· Vision · vision-language, transformer

Paper: Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Swin Transformer: a hierarchical Transformer with Patch Merging and shifted windows

ViT patches images and applies global self-attention for global modeling, but this comes at the cost of weaker multi-scale representation. Multi-scale features matter for downstream CV tasks. Swin performs local attention within windows and uses shifted windows to enable cross-window interaction, approximating global modeling while saving memory and achieving strong performance.
Toward a unified CV–NLP modeling paradigm, ViT is more general: it relies on little inductive bias and performs well in both domains without extra structure. Swin incorporates more prior knowledge, analogous to the sliding-window design in CNNs.
Patch Merging: an operation similar to strided convolution for downsampling feature maps.
Shifted windows: window-based self-attention together with shifted-window self-attention reduces computation while improving global receptive field.
Efficiency of shifted-window attention: masked attention; relative positional encoding instead of absolute positional encoding.

5.1 After window shifting, the number of patches can increase; patches of inconsistent spatial layout cannot be batched directly, whereas zero-padding implicitly adds unnecessary computation.

5.2 The method partitions and pads regions, applies masked MSA so patches assigned to the same partition do not attend across partition boundaries, then reverses the cyclic shift—avoiding unbounded spatial drift of the feature map.