Controlled Entity-Centric Summarization Large Language Model
Abstract
Summarizing long text or corpora of texts isa critical use-case for Natural Language Processing (NLP)technology. It allows readers to parse key information frommore source documentation than they otherwise could have read in a given amount of time. In contrast to generic methods, a summary that can be constrained to specific content of interest to a reader extends the usefulness of summary methods to scenarios where the entire source document is not relevant.This can be referred to as a controlled summary. This paper proposes multiple methods to use Large Language Models(LLMs) for producing summaries focused on specific entities found in a text. Our best-performing method, which we term Controlled Entity-centric Summarization LLM (CESL), uses aninstruct-tuned LLM (GPT-4 Turbo) and outperforms previous state-of-the-art approaches without additional fine-tuning. We include many additional experiments in this paper to act as an applied survey for various prompting and generation strategies in the task of entity-centric summarization. We propose additional metrics for abstractive summarization performance beyond commonly used summarization metrics such as ROUGEor BERTscore and demonstrate a framework for how they can be used to identify problematic results responsibly and proactively,allowing a human in the loop to focus on review of high-payoff results. We publicly release an improved version of the entSUM benchmark dataset. We also extend the findings of previous work regarding limitations of prompting LLMs to show that given certain prompts, some LLMs will default to answers found in their parametric memory even if explicitly instructed to rely on retrieved context.1
Document Details
- Document Type
- Technical Report
- Publication Date
- Jan 22, 2024
- Accession Number
- AD1225611
Entities
People
- Alex E. Lichtenberg
- Lei Hamilton