Tuning Abstractive Summarization Models Towards Increased Novelty

Abstract

Neural machine translation models, based on attention and pointer-mechanism, has in recent studies been successfully applied to the task of Abstractive Summarization of long documents such as news articles. Although state-of-the-art architectures inhibit abstractive capabilities, it has been observed that these models mostly copy large fragments from the source, even in scenarios where the models should paraphrase and use novel word combinations.

In this thesis we explore the possibility of improving the novelty in the model generated summaries. After training strong baseline models by combining architectural components from state-of-the-art systems, we attempt to improve the novelty by (1) selective data sampling, (2) adding a novel extraction loss component and (3) by engineering reward functions that captures novelty used in optimization by reinforcement learning. We explore multiple parameters related to each approach, and present quantitative scores in terms of relative ROUGE increase and qualitative output from each model.

For our reinforcement learning experiments we demonstrate higher ROUGE scores compared to previous work utilizing joint policy gradient loss and single model architecture. However the textual quality of our is left to be determined.