ROBUST MULTIMODAL MACHINE LEARNING FOR INTERVIEW SKILL ASSESSMENT AND GENERATING EXPLAINABLE FEEDBACK
Abstract
Nowadays, social communication skills (CS) are required in various situations, such as business, education, mental health counseling, etc. [1]. Though younger people need the training opportunities to obtain sufficient CS, they don’t always have the training opportunities, and it isn’t easy to understand their current skill level. It is essential to implement the CS training system that can automatically assess the current level of CS of users by sensing their verbal and non-verbal information and providing feedback to improve CS. It is a crucial interdisciplinary challenge across social psychology, human-computer interaction, and multimodal machine learning. Toward implementing the automatic CS training system, the first core technique is a machine learning algorithm for accurate and robust prediction of CS from various multimodal features of speakers. Observable multimodal features include visual (gaze, gesture, and facial expression), acoustic (prosody, pose, fluency, speaking speed) and linguistic information (vocabulary, logical aspect) and physiological (heart rate, EDA) information. The second core technique is to explain why they are judged with the score of CS and to generate feedback to improve the skills.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Apr 20, 2023
- Source ID
- FA23862214034
Entities
People
- Shogo Okada
Organizations
- Air Force Office of Scientific Research
- Japan Advanced Institute of Science and Technology
- United States Air Force