Transfer Learning with Real-World Nonverbal Vocalizations from Minimally Speaking Individuals
Abstract
We trained and evaluated several types of transfer learning to classify affect and communication intent of nonverbal vocalizations from eight minimally speaking individuals (mv*). Datasets were recorded in the real-world with in-the-moment labels from a close family member. We trained deep neural nets (DNNs) on six audio datasets (including our dataset of nonverbal vocalizations) and then fine-tuned the models to classify affect and intent for each individual. We also evaluated a zero-shot approach for arousal and valence regression using an acted dataset of nonverbal vocalizations that occur amidst typical speech. For two of the eight mv* communicators, fine-tuning improved model performance compared to fully personalized DNNs and there were weak groupings in arousal values inferred using zero-shot learning. The limited success of the evaluated transfer learning approaches highlights the need for specialized datasets with mv* individuals.
Document Details
- Document Type
- Technical Report
- Publication Date
- Jul 23, 2021
- Accession Number
- AD1188944
Entities
People
- Jaya Narain
- Kristina T. Johnson
- Pattie Maes
- Rosalind Picard
- Thomas F. Quatieri
Organizations
- Harvard University
- MIT Lincoln Laboratory
- Massachusetts Institute of Technology