Modeling, Computations, and Applications in Multimodal Information Integration
Abstract
Statement Of Work:This project develops fundamental mathematical and computational theories to model and integrate complexmultimodal data and information, from video to faces and to networks. Developing and combining advanced theoretical and computational tools in random forests, subspace learning, model aggregation, compressed models, signal processing of and on networks, and multimodal data analysis, the PI will address challenges in multiple applications. These include low-cost and computationally efficient multimodal face recognition, multimodal network inference and analysis, cross-modality retrieval, and multimodal detection combining video and text, to name a few of the addressedapplications. The work considers limited training data availability, a common scenario in Department of Defense applications. For this, theories and practices of both training and testing data augmentation are investigated. With the developed framework a reduction of several orders of magnitude in labeled data and computational cost is achieved when compared with alternative approaches. The underlying approaches, which also formally address the robustification and metric components of learning, provide instruments for analyzing and improving other techniques such us deep learning. The project builds on extensive collaboration with the Department of Defense and combinestheory, computational tools, and applications, to advance in the area of multimodal data integration.Objective:This project deals with the modeling and integration of multimodal high-dimensional data. In the era of big data, one might first wonder if multimodal data can provide any type of information that can~t be obtained with a single modality. The PI has recently addressed this question, and formally showed the answer to be yes, and even detailed cases when multimodality is needed and when it is not. Beyond such important foundations, multimodal data is critical when data is scarce, noisy, and uncertain, as it is common in the Department of Defense applications, challenges often ignored inthe current big data research emphasis. Such scenarios motivate the proposed work. Different modalities can help to do joint inference, as is demostrated in the network analysis case. Moreover, often the data available for training is from a very different modality than the one for testing, as in the case of infrared face recognition discussed in this project. In this regard, multimodality can help in data augmentation, a criticalcomponent to be studied here as well for improving learning and classification. Finally, connections between modalities goes beyond recognition and classification, and inferring a condition from unexpected data sources, is of paramount importance in disciplines ranging from marketing to health-care and defense.Approach:The exploitation of multimodal data is one of the unifying themes of this project. It should be noted that in this project the PI aims to consider both usual and unusual sources of multimodal data, including but not limited to airborne data, ground data, RGB and infrared data, tweeter, Internet traffic, images and videos from the media, and audio. A second unifying theme of this project is the underlying mathematical foundation is subspace modeling. These mathematical tools have been demonstrated by the PI and others to be very powerful for theoretical analysis and stateof-the-art practical applications. The PI develops and brings tools from subspace modeling in the form of learning multimodal low-rank representations, modeling multimodal sparse networks, and solving for big data matrix decompositions. Such models form also the basis for some of his theoretical studies. Finally, all the work has efficiency behind it, this being manifested from the development of memory and computational efficient hashing forests to the development of a novel algorithm for non-negative matrix factorization. A third unifyingmotif of this work is then the ubiquitous consid
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Aug 08, 2016
- Source ID
- N000141612343
Entities
People
- Guillermo Sapiro
Organizations
- Duke University
- Office of Naval Research
- United States Navy