Visual Analytics for Threat Action Detection, Precognition, and Justification
Abstract
The primary objective of this project is to explore a visual precognition framework that effectively provide risk assessment for thr,eat actions. We plan to develop Threat Action Graphs (TAGs) over large-scale video data for representation learning. The representat,ions will enable from learning from complex and dynamic environments, where a variety of vision tasks can be achieved, including ano,maly detection and prediction, uncertainty modeling, few-shot learning, video captioning, model explanation, etc. The proposed frame,work will achieve the following three properties: (i) Exploiting the hierarchical structure of the dynamic scene for learning effect,ive representations to capture complex interactions between humans and objects in battlefield scenes. (ii) Enabling risk assessment,ity using external knowledge to build trustworthy models. We will create a novel trustworthy framework that allows the human decisio,n team to better understand model?s outputs. All of these properties enable threat analysis to support mission operations. The DURIP, instrumenta, the way for visual precognition. The proposed project will advance machine learning and computer vision in the following key areas,: dynamic graph modeling, intention prediction, event detection, few-shot learning, dense captioning, explainable AI, and uncertaint,y modeling. More specifically, it extends the state-of-the-art in three main directions: 1) developing novel dynamic hierarchical Th,reat Action Graphs for capturing complex interactions between humans and objects in battlefield scenes; 2) developing probabilistic,multiple instance learning and sequential visual forecasting for present and future risk modeling, respectively, with uncertainty-ba,sed model adaptation from limited data; 3) Graph-based counterfactual visual explanation and video captioning using external knowled,ge for justifying models decisions. These technical innovations will be elegantly integrated into the proposed framework for visual, precognition of threat actions. The practical goal of our proposal is to build a visual precognition system for threat action unde,rstanding system through the powerful GPU cluster. This new research endeavor will equip DOD with a new capability of visual precogn,ition for future events. A key issue for DOD in developing situational awareness, which tends to detect after sudden attacks causing, hundreds of deaths and injuries. By comparison, the proposed visual precognition system will raise an alarm before the attack happe,ns. Such a new capability allows the visual intelligence system to peek into the future, and further enhances its usefulness in miss,ion operations. To build human-machine trust, the proposed system provides visual and textual explanation and justification for visu,al precognition. This supports models decisions on suspicious motion imagery and is closely and timely relevant to the DOD s missio,n. In addition, this new capability of visual precognition is adaptive, and can foresee novel events using limited training data, wh,ich is useful in dynamic battlefields. All of these new features are grounded upon the proposed hierarchical threat action graphs. T,hese powerful graphs are capturing the static and dynamic properties of visual entities and their interactions. They are optimally i,mplemented to sense, interpret, and reason in an open world with large-scale unconstrained real-time visual data, achieving operatio,nal capabilities, such as risk modeling and Persistent Pervasive Tactical Surveillance. This perfectly fits the needs of DOD. Approv,ed for Public Release
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Dec 06, 2022
- Source ID
- N000142312046
Entities
People
- Yu Kong
Organizations
- Michigan State University
- Office of Naval Research
- United States Navy