Multi-Turn Generation-Based Conversational Agents in Open Domains

Abstract

This PhD thesis focuses on open-domain generation-based conversational agents, which are chatbots that generate responses to any input or question using natural language processing and deep learning techniques. The thesis identifies three major challenges faced by these conversational agents.

(1) Generating appropriate responses for a wide range of topics and domains. Current studies have focused on single-corpus training, which limits the model's ability to generate relevant responses for certain topics.

(2) Improving a model's performance of context attention distribution in multi-turn settings. The ability to distribute attention and assign importance to relevant information is necessary to generate appropriate responses. However, most existing works have treated multi-turn conversations as one-turn contexts, limiting the performance of the agents.

(3) Integrating knowledge under the conversational question-answering task perspective. There is a gap in research on integrating extractive question-answering techniques with instruction-based tuning and prompt-based tuning. The thesis proposes several approaches to address these challenges.

For (1), the thesis proposes Document-specific Frequency (DF) as an evaluation metric and proposes several methods for balancing multiple corpora. The best method, which integrates DF with the training, achieves an improvement by 34.1% on F1 performance and at least 20.0% on DF. A thorough human evaluation shows a highly significant (p < 0.001) improvement in all of our proposed methods.

For (2), the thesis proposes Distracting Attention Score ratio (DAS ratio) as an evaluation metric and employs self-contained negative samples and summarization techniques to improve a system's performance on context attention distribution. The proposed self-contained negative samples are applied as a training strategy, resulting in about 10% better DAS ratio. The best summarization technique setting with ORACLE gains a 23% improvement on the DAS ratio.

For (3), the thesis explores various settings of integrating extractive question answering with instruction-based tuning, prompt-based tuning, and multi-task learning. When combining prompt-based tuning with either instruction-based tuning or multi-task learning, the F1 performance is improved by about 18% over the baseline.

Together, these techniques have improved the overall performance of multi-turn conversational agents on open domains.