Application of Algorithm Learning to Identify and Mitigate Concept Drift
Abstract
Data streams are becoming more numerous and complex, driven by an increased number of capable sensors. The complex, highly dimensional datasets created by these sensors contain information critical to our understanding of the battlefield situation. A significant change in the adversary's tactics, techniques, and procedures (TTPs) leads to a shift in the collected sensor data. The shift in the distribution of features in the data stream is known as concept drift, and this drift can be detected through predictive machine learning. We propose a novel drift detection method, named Reduced Dimensionality Drift Detection (RD3), based on dimensionality reduction to decrease feature space through supervised learning and unsupervised detection. Additionally, we show that concept drift can be mitigated after detection via an automated algorithm. We validate the performance of our novel method through comparison to a proven detection method, in similar trials and conditions, and show that RD3 performs comparably in concept drift detection and mitigation in all datasets evaluated.
Document Details
- Document Type
- Technical Report
- Publication Date
- Mar 01, 2021
- Accession Number
- AD1150410
Entities
People
- Nicholas T. Balk
Organizations
- Naval Postgraduate School