ToMCAT: Theory of Mind-based Cognitive Architecture for Teams
Abstract
Current state-of-the art approaches to designing artificially intelligent (AI) teammates address some of the capabilities that they must have, including inferring the internal states of other agents, solving problems collaboratively with them, and communicating with them in a socially-aware manner. To be a truly effective teammate, an AI agent must possess all of these capabilities, but no current approach combines them. To address this, we propose a Theory of Mind-based Cognitive Architecture for Teams (ToMCAT) with these capabilities. ToMCAT is comprised of a set of local agents (one for each human teammate) equipped with cameras and microphones to capture facial expressions and speech, as well as virtual sensors that record the local environment within the ASIST virtual testbed, the actions performed by human teammates, and chat exchanges between them. The local agents communicate with their respective humans, as well as with a global agent that performs coordination and global team optimization. ToMCAT is built around the following fundamental research efforts: Multi-modal dialogue system: We will develop algorithms to construct rich representations of the environment, human teammates, and their interpersonal relationships using data from physical and virtual sensors. These include multi-modal sentiment analysis and coreference resolution to infer latent beliefs, desires, and intentions of human teammates. We will also extract task and goal specifications from dialogue between teammates, for automated planning. By learning a mapping between high-level actions and the low-level task primitives observed using software sensors, ToMCAT will learn the sequences of actions required to execute interventions in the virtual testbed. Grounded probabilistic modeling of teams: We will develop an interpretable probabilistic modeling framework for modeling interpersonal dynamics between team members, as well as overall latent properties of the team itself, such as coordination. Additionally, we will simultaneously analyze the brain activity of multiple interacting humans with their audio and video streams to acquire ground-truth information about the quality and strength of interpersonal coordination in teams and use it to greatly enhance learning about the internal states of the team members from the ASIST sensor suite, which we expect to be comprised of inexpensive and potentially noisy sensors. The modeling framework and the data from these grounding experiments will be made freely available to all performers and serve as valuable program resources. Decentralized planning and socially-aware interventions: We will implement a decentralized, assumption-based planning framework that is robust in the face of partial information and noisy communication channels. Background domain knowledge will be combined with candidate tasks and goals inferred from observation and dialog, as well as inferred latent mental states of human teammates, to enable ToMCAT to perform focused, opportunistic planning to maximize team member s coordination and social engagement relative to overall team goals. This includes implementing a natural language generation module that will enable the local agents to communicate optimal high-level strategies to the human teammates. Our approach represents a novel synthesis of research at the frontiers of social, affective and cognitive science, artificial intelligence, and natural language processing, and dramatically advances the state-of-the art in all of these fields.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Dec 04, 2019
- Source ID
- W911NF2010002
Entities
People
- Adarsh Pyarelal
Organizations
- Army Contracting Command
- Defense Advanced Research Projects Agency
- University of Arizona