Visual Analytics for Threat Action Detection, Precognition, and Justification
Abstract
ONR Program Officer: Dr. Behzad Kamgar-Parsi, Machine Learning, Reasoning and IntelligenceARO TPOC: Dr. Hamid Krim, Computer Science,, Information SciencesThe,isk assessment for threat actions. We plan to develop Threat Action Graphs (TAGs) over large-scale video data for representation lea,rning. The representations will enable from learning from complex and dynamic environments, where a variety of vision tasks can be a,chieved, including anomaly detection and prediction, uncertainty modeling, few-shot learning, video captioning, model explanation, e,tc. The proposed framework will achieve the following three properties: (i) Exploiting the hierarchical structure of the dynamic sce,ne for learning effective representations to capture complex interactions between humans and objects in battlefield scenes. (ii) Ena,bling risk assessment for both present and future events with few-shot learning capability. (iii) Effective video captioning and mod,el explanation capability using external knowledge to build trustworthy models. We will create a novel trustworthy framework that al,lows the human decision team to better understand models outputs. All of these properties enable threat analysis to support mission,ove the correctness of the way for visual precognition.The proposed project will advance machine learning and computer vision in the, following key areas: dynamic graph modeling, intention prediction, event detection, few-shot learning, dense captioning, explainabl,e AI, and uncertainty modeling. More specifically, it extends the state-of-the-art in three main directions: 1) developing novel dyn,amic hierarchical Threat Action Graphs for capturing complex interactions between humans and objects in battlefield scenes; 2) devel,oping probabilisticmultiple instance learning and sequential visual forecasting for present and future risk modeling, respectively,,with uncertainty-based model adaptation from limited data; 3) Graph-based counterfactual visual explanation and video captioning usi,ng external knowledge for justifying models decisions. These technical innovations will be elegantly integrated into the proposed f,ramework for visual precognition of threat actions.The practical goal of our proposal is to build a visual precognition system for t,hreat action understanding system through the powerful GPU cluster. This new research endeavor will equip DOD with a new capability,of visual precognition for future events. A key issue for DOD in developing situational awareness, which tends to detect after sudde,n attacks causing hundreds of deaths and injuries. By comparison, the proposed visual precognition system will raise an alarm before, the attack happens. Such a new capability allows the visual intelligence system to peek into the future, and further enhances its u,sefulness in mission operations. To build human-machine trust, the proposed system provides visual and textual explanation and justi,fication for visual precognition. This supports models decisions on suspicious motion imagery and is closely and timely relevant to, the DOD s mission. In addition, this new capability of visual precognition is adaptive, and can foresee novel events using limited,training data, which is useful in dynamic battlefields. All of these new features are grounded upon the proposed hierarchical threat,y are optimally implemented to sense, interpret, and reason in an open world with large-scale unconstrained real-time visual data, a,chieving operational capabilities, such as risk modeling and Persistent Pervasive Tactical Surveillance. This perfectly fits the nee,ds of DOD.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Apr 01, 2022
- Source ID
- N000142212309
Entities
People
- Yu Kong
Organizations
- Office of Naval Research
- Rochester Institute of Technology
- United States Navy