Reliable Multimodal Data Fusion from Physical and Human Sensors with Quality Assurance
Abstract
The overall research goal of this project is to build a Reliable Multimodal Data Fusion Engine (RMDFE) to efficiently fuse the sensory data from both physical and human sensors and rigorously quantify the accuracy of the fusion results under a unified analytical framework. The project closely fits the direction of Data and Information Fusion in the ÒInformation Processing & FusionÓ program at the Army Research Office (ARO). The research is motivated by the observation that the physical world is being increasingly sensed and monitored by multiple sensor data streams sourced from both physical sensors (e.g., cameras, radars, mobile phones) and human sensors (e.g. Twitter, Facebook). The disparate sensing capabilities of physical and human sensors provide the great opportunity for them to work together in a complementary way to obtain an augmented state of the physical world. Therefore, reliably integrating Human Intelligence (HUMINT) and Signal Intelligence (SIGINT) in the data fusion process is highly desirable as it will offer powerful understanding of complex environments and situations, but this goal has yet to be achieved. The project has four objectives to achieve the overall research goal: 1) represent the sensor measurements from both physical and human sensors using a unified data representation and generate the meta-data for the fusion process; 2) establish an Assured Physical-Human-Sensor Fusion Model (APFM) to model the unknown source reliability and rigorously assess the quality of the fusion results using a principled analytic framework; 3) explicitly address the dependency between data sources and the correlation between measured variables in APFM; 4) seamlessly integrate the data pre-processing components and APFM to build a Reliable Multimodal Data Fusion Engine (RMDFE) and validate its performance in a real-world multimodal sensing application. In the proposed solution, a new estimation theoretic approach will be developed to jointly estimate the source reliability and correct values of measured variables and compute the accuracy bounds of the estimation results. The proposed research fills in the key knowledge gap of current data fusion solutions, which is the lack of principled approaches to accurately model the unknown source reliability from the physical and human sensor data and rigorously quantify the quality of the analysis results. This project is among the pioneering work to develop a new sensing and fusion system where both physical and human sensor data are fused under a unified framework with performance guarantees. The results of the project will directly contribute to the militaryÕs intelligent decision-making process by offering a new reliable multimodal data sensing and fusion system (i.e., RMDFE) that explores the collective power of physical and human sensors. The RMDFE can be applied to enhance situation awareness for various tasks from identifying, discriminating and reacquiring targets to understanding the surveillance of patterns and interaction between entities. This YIP project serves as a critical step towards achieving one of the PIÕs long-term research missions, which is to develop novel analytic foundations for a new science of summarizing information from a large collection of heterogeneous, interdependent, and potentially unreliable data sources, and to build the next generation of data fusion and information distilling service that extracts reliable information with quality guarantees. The success of the project could lead to future reliable cyber-physical-human systems that explicitly incorporate source reliability modeling and fusion results assessment into a rigorous analytical framework. The PI has a strong expertise and extensive experience in data analytics, information fusion, estimation theory and multimodal sensing applications, which are essential to the success of the project.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Oct 15, 2018
- Source ID
- W911NF1710409
Entities
People
- Dong Wang
Organizations
- Army Contracting Command
- United States Army
- University of Notre Dame