ACL-2018 Improving Abstraction in Text Summarization
Decomposing the Decoder into a Context Network and a Pretrained Model
Abstract
Among existing methods, accurate abstractive summarization—as measured by novel phrases that do not appear in the source document—remains low. This paper proposes two ways to improve the level of abstraction in generated summaries. The first decomposes the decoder into a context network that retrieves relevant information from the source document and a pretrained model; the second is a new metric
Introduction
The first contribution reduces the decoder’s extractive and generative responsibilities by splitting it into a context network and a language model: the context network compresses the source document, and the language model produces concise paraphrases.
The second contribution is a hybrid objective that jointly optimizes n-gram overlap with ground truth while encouraging abstraction.
Model
Base Model and Training Objective
The encoder uses a biLSTM.
Temporal attention context score computation
Intra-attention context computation during decoding
Probability of generating from the output vocabulary
Probability of selecting a word from the fixed vocabulary
Probability of copying from the source document
Maximum-likelihood training objective
Policy learning uses ROUGE-L as the reward and greedy decoding as the self-critical baseline
The final loss is a mixture of the two
Language Model Fusion
An additional benefit of this decomposition is that external knowledge about fluency and domain-specific style can be incorporated easily by pretraining the language model on a large text corpus.
A 3-layer LSTM is used; the hidden state of the last LSTM layer of the language model is fused with Equation (8).
$g_t$ is the gating function
Replace the output distribution of Equation (10) with
Abstract Reward
See the original paper.
Experiment
<HR align=left color=#987cb9 SIZE=1>