Event Detection for Streaming Analytics: An Intelligent Mathematical Paradigm
Abstract
Detection of anomalous and/or novel events over high-velocity, dynamic, and unbounded streaming data is a fundamental yet highly demanding research problem in statistical pattern recognition and big data analytics. It is common and of paramount importance to many scientific and application domains, such as astronomy, security and military ISR operations, where devastating consequences can be caused if the anomalous items and/or new signals are not detected promptly. To date, real-time event detection has been the subject of various statistical reasoning and machine learning research. However, the bulk of existing methods often treat ÒanomaliesÓ and ÒnoveltiesÓ equivalently, lacking effective and efficient algorithmic mechanisms to distinguish these two types of events on the fly while integratively handling the unique challenges that streaming data imposes on them. In this DoD project, we propose to create an innovative, reliable and scalable event detection prototype with theoretical guarantees for streaming data. The proposed work is logically built on our prior productive research in data streams. It will begin from conceptually differentiating ÒnoveltiesÓ from ÒanomaliesÓ, then proceed to substantially extending our newly-developed streaming data mining system to real-time event detection, and further creating advanced detection algorithms for more pressing yet understudied challenges in mining streaming data. To this end, we will develop the following key techniques. 1). An adaptive online kernel density estimation based algorithm to accurately pinpoint isolated anomalies and cohesive novel patterns from unlabeled, concept-drifting data streams with noise; 2). A dynamically evolving recurrent neural network to reveal suspicious rare events either as a semi-supervised model in a finite label latency context or as a supervised model in an infinitely delayed label scenario; 3). An online margin-based learning method to effectively handle evolving feature spaces while performing scalable interpretable event detection over noisy streaming data without expert labels; and 4). A multi-task learning framework to collaboratively conduct reliable and stable event detection when facing multi-source asynchronous raw data streams. All these advances will be embodied in an intelligent mathematical paradigm that will offer big data practitioners an open-source toolbox of qualitatively different approaches. The proposed research will advance the fundamental real-time anomaly detection and novel pattern discovery by learning more distinguishable and reliable patterns, modeling the effects and relevance of involved data sources, and leading analystsÕ attention to the suspicious and unseen events in massive streaming data. The ability to learn, distinguish and characterize anomalous and novel signals in (multiple) streaming data will enhance the effectiveness of online risk and threat assessment, prevention, and neutralization, enabling the DoD to renovate a considerable knowledge-centric real-time flow of information to assist in critical strategic decision making. Event detection over streaming data spans many application domains that are important to the military and national interests. This project will invent next-generation real-time event detection techniques that have the potential to impact these fields as well. This project will also enhance Xavier University of Louisiana (XULA)Õs research capabilities and competiveness in big data analytics, machine learning, high-performance/mathematical computing, and related interdisciplinary informatics fields. XULA undergraduate students, those historically under-represented in STEM, will have vital opportunities to participate in course-based and extracurricular research, thereby preparing them for STEM-specific opportunities in graduate schools and future professional careers.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Jul 09, 2020
- Source ID
- W911NF2010249
Entities
People
- Kun Zhang
Organizations
- Army Contracting Command
- Office of the Secretary of Defense
- Xavier University of Louisiana