Multimodal Subspace Learning and Modeling of Complex Systems
Abstract
Major Goals: This project deals with the modeling of multimodal data. In the era of big data, one might ask if multimodal data can provide any type of information that cant be obtained with a single modality. We recently addressed such question, and formally showed the answer to be yes, and even detailed cases when in theory multimodality is needed and when it is not. Beyond such important foundations, multimodal data is critical when data is scarce, noisy, and uncertain, as it is common in the Department of Defense applications, challenges often ignored in the current big data research emphasis. Different modalities can help to do joint inference, as for example here addressed in the network analysis case. Another example of the potential of multimodal data is our recent work for arousal and emotion analysis, where the combination of multimodal images allows to produce at extremely low-cost results that were possible only with high-end devices before. Moreover, often the data available for training is from a very different modality than the one for testing, as in the case of infrared face recognition discussed in this project. Finally, connections between modalities goes beyond recognition, and inferring a condition from unexpected data sources, as we recently did for individual emotion analysis via tweeter, is of paramount importance in disciplines ranging from marketing to health care and defense. The exploitation of multimodal data is therefore one of the unifying themes of this project. We should note that in this project we want to consider both usual and unusual sources of multimodal data, including but not limited to airborne data, ground data, RGB and infrared data, tweeter, Internet traffic, images and videos from the media, and audio. Moreover, with a bit of abuse of notation, we consider multiple instances of the same modality, e.g., population studies, as multimodal.
Document Details
- Document Type
- Technical Report
- Publication Date
- Dec 20, 2019
- Accession Number
- AD1097126
Entities
People
- Gillermo Sapiro
Organizations
- Duke University