ICML-2019 Parameter-Efficient Transfer Learning for NLP
adapter-bert: Small modules embedded in the transformer
Abstract
Fine-tuning large pretrained models is a standard transfer-learning approach in NLP, but for many downstream tasks it is parameter-inefficient: each task requires an entirely new model. This work proposes transfer via adapter modules: only a small set of trainable parameters is added per task while the rest of the network stays fixed, enabling heavy parameter reuse. To validate adapters, BERT is adapted to 26 text classification tasks. Experiments show that tuning only a small fraction of parameters can match full fine-tuning of all parameters.
Adapter tuning for NLP
Here an adapter is a down-projection FC layer; each transformer block contains two adapters, the remaining layers are frozen, and only the adapters are trained.
Experiments
<HR align=left color=#987cb9 SIZE=1>