Assessing the Efficacy of Face-to-Face Communication: Towards Improving Human-Machine Communication
Abstract
The central problem of human face-to-face (f2f) communication is that communicative actions (e.g., facial displays, gestures, and even speech) are always ambiguous to the extent that their meaning depends on the shared knowledge of the agents. The ambiguous nature of communicative actions requires that the interlocutors, including artificial agents, align their conceptual space (i.e., what they specifically mean by their verbal and nonverbal communicative actions). However, conceptual alignment cannot be directly measured, and theoretical descriptions lack a testable explanation of what drives corrective actions when conceptual alignment is imperfect.Our novel theory aims to fill these gaps by suggesting that during f2f communication, interlocutors strive to elicit predictable responses from each other. If supported by the data, this theory opens a clear pathway towards applications in human-machine communication and training.Current theories of brain function propose that the general function of the nervous system is to predict future states of the environment. Within this framework, we argue that successful f2f communication also relies on correctly predicting the partner#s behavior. Predictable responses indicate understanding, while unpredictable ones may result from lack of understanding, theneed to introduce a new aspect to (novel information) or steer the conversation away from the current topic (unwillingness to deal with the issue).A sufficiently large shared conceptual space is also hypothesized to give rise to behavioral and brain-to-brain coupling, which are often observed in f2f communication. We propose to capture the dynamics of conceptual alignment by detecting markersof successful and unsuccessful predictions. We assume that a) the frequency of failed predictions is inversely related to communication success, b) prediction errors lead to decreased brain-to-brain and behavioral synchrony, and c) prediction errors are accompanied by characteristicchanges in brain networks and behavioral patterns.We will test our hypotheses in ecologically valid communication scenarios varying in complexity and by manipulating predictability within the scenarios. During the interactions, we will continuously measure behavioral (gaze and body movement, pupil size, prosodic and semantic characteristics of verbal communication) and brain signals (EEG) and compare their synchronization between the interlocutors with objective (an AI language model trained on conversations) and subjective (provided by the participants) measures of predictability assessed at least for each turn in the interaction. This will allow us to dynamically follow the build-up of shared understanding during f2f communication and to determine the patternsof neural and behavioral signals accompanying successful and unsuccessful predictions.The variety of scenarios and measurements allow tailoring the approach to several different applications including human-machine communication and training. We believe that a main obstacle on the way to artificial agents becoming capable of meaningful communication with humans lies in their lack of in-depth understanding of the development of a shared conceptual space. Utilizing online markers of understanding/agreement, an artificial agent could engage in corrections to restore/improve shared understanding. Moreover, these measurements would allow on-line monitoringof training by communication as well as helping to devise training courses aimed at improving communication skills.The proposed work aligns with the goals of Office of Naval Research Human and Bioengineered Systems Division and will contribute to Warfighter Supremacy by enabling a more trained and ready force (augmented warfighter), which will be able to quickly adapt to future changes in training requirements and at the field of operations.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Apr 12, 2023
- Source ID
- N629092312025
Entities
People
- Istvn Winkler
Organizations
- Office of Naval Research
- United States Navy