Communication through Gestures, Expression and Shared Perception
Abstract
The Colorado State University effort as part of the DARPA Communicating with Computers (CwC) program emphasized peer to peer communication between a person and an embodied agent both of whom were engaged in solving a concrete physically paced problem. Early in the program, Colorado State University teamed up with Brandeis University and collectively created what is described by us as the Diana System. The Diana System set new performance standards for combined verbal and nonverbal communication between people and agents. It demonstrated new capabilities for asynchronous multi-modal interaction. It fostered fundamental advances in AI representational tools in order to support peer-to-peer communication grounded in shared perception. From the standpoint of Colorado State University, the description of what was accomplished was the creation of an embodied agent able to conduct peer to peer communication using a combination of sight and speech; in particular, the key is tight coupling of computer vision to observe the person and seamlessly combine verbal and non-verbal communication. The next section will summarize our approach and significant accomplishments. It will highlight three key contributions of the work. The critical decision early on to carefully study how two people communicate when solving a visual task. The integration of sight, speech and most fundamentally a shared perception of a physical CSU Communication task environment. It will be essential to understand we are describing the joint work of Brandeis University and Colorado State University. Finally, there is the discovery that multimodal communication is fundamentally different from unimodal in many ways.
Document Details
- Document Type
- Technical Report
- Publication Date
- Nov 30, 2021
- Accession Number
- AD1200466
Entities
People
- Francisco R. Ortega
- J. R. Beveridge
Organizations
- Colorado State University