Givenness Hierarchy Theoretic Natural Language Generation for Situated Human-Robot Interaction
Abstract
Robots teaming with humans must be able to concisely, naturally, and effectively refer to the entities in their environment. Crucially, this requires choosing both referring form (e.g., definite description, anaphora, and/or deixis) and referring content (i.e., what properties to use in descriptions). We propose to achieve these capabilities using a rich Givenness Hierarchy theoretic model of cognitive status, informed by both linguistic and extralinguistic cues, and modeled using a combination of knowledge- and data-driven methods. We further propose to use this model to better select referring forms and referring content. To do so, we will execute fourteen research tasks in service of seven research questions organized into three research thrusts.In year one, we will pursue an experimental research thrust. To determine how ground truth data regarding cognitive status be collected, we will use a novel experimental paradigm to collect and annotate a collection of dyadic human-human interactions to include information regarding presumed cognitive status. To determine how extralinguistic status-relevant information can be extracted from multisensory observations, we will develop multisensory models of sequential behavior recognition that combine regularized optimization with constrained optimization. In year two, we will pursue a modeling research thrust. To determine how cognitive status can be computationally modeled, we will investigate both data-driven modeling techniques (using Dynamic Bayesian Networks) and knowledge-driven modeling techniques (using Satisfiability Modulo Theories), and then combine these techniques into a unique hybrid model that leverages the strengths of both approaches. To determine how language can be used to inform mental representations, we will develop a zero-shot learning approach to learning representations of objects from natural language that enables optimal subsequent visual identification and tracking.In year three, we will pursue a communication research thrust. To determine how cognitive status can be leveraged during the selection of referring forms, we will investigate both data-driven modeling techniques (using data-driven constrained rank trees) and knowledge-driven modeling techniques (using constrained optimization), and then combine these techniques into a unique hybrid model that leverages the strengths of both approaches. To determine how cognitive status can be leveraged during the selection of referring content, we will investigate two techniques: an incremental approach, in which we extend the classic Incremental Algorithm to separately consider both egocentric and allocentric generation factors, and a unified approach, in which both types of factors are considered within a single constrained optimization framework. To determine how representations can be learned that enable optimal selection of referring content, we will develop a novel adaptive discriminative representation learning approach that promotes optimally distinguishing attributes during the learning process, and show how this aplection.Our proposed work is expected to advance the state of the art in natural language processing and robotics, and provide insights to cognitive psychologists and linguists. Furthermore, it is expected to impact future navy relevance by enhancing future warfighter performance through intelligent interactive robotic teammates; by analyzing and replicating cognitive mechanisms, this research will provide a better understanding of what makes humans unique and effective, and improve the effectiveness of both autonomous agents and their human teammates.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- May 05, 2021
- Source ID
- N000142112418
Entities
People
- Thomas V. Williams
Organizations
- Colorado School of Mines
- Office of Naval Research
- United States Navy