Human-aware probabilistic logic approach for learning with less labels from multimodal data to model credibility
Abstract
The goal is to address the challenge of generating multiple hypotheses from a common semantic representation that is constructed from multiple modalities such as text, vision and speech. Our specific goal is in the context of conversational AI where the goal is to identify specific anomalies from a combination of videos, audio and text data. The system should be able to model uncertainty and noise in the schema obtained from low level data. There is a necessity that the proposed system must be able to represent and reason with complex, noisy data. Finally, this system must be able to generate multiple interpretable hypotheses that can be explained to a human expert in an effective manner. To this effect, we propose an integrated system that generates explainable hypotheses from multiple disparate sources. Our proposed system has three distinct capabilities that makes it attractive for large-scale inference tasks. First is the capability of the system to handle noisy, potentially inconsistent, and even incorrect inputs from multiple sources. This is due to the fact that each system could be parsed independent of each other and thus generate specific noise. Second is the ability of the system to reason at a higher level, i.e. on the level of objects and relations that allow for generalized hypothesis generation that can be applied and specialized as necessary. Last, and perhaps most important feature of the proposed system, is the capability of interacting with human experts in a natural and seamless manner, allowing for rich inputs from the human experts. In our system, the human expert can provide guidance at every step of the pipeline. While existing systems require significant engineering, our system is fully automatic and lets the system interact with the human by generating human-interpretable hypotheses that can be refined and improved as necessary.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Feb 29, 2024
- Source ID
- FA95502310239
Entities
People
- Sriraam Natarajan
Organizations
- Air Force Office of Scientific Research
- United States Air Force
- University of Texas at Dallas