Anomaly Detection Using A Variational Autoencoder Neural Network With A Novel Objective Function And Gaussian Mixture Model Selection Technique
Abstract
Anomalies in data often convey critical information that can be leveraged in a variety of applications. For the military engaged in combat, this can amount to identifying threats early and preserving a lethal edge over an adversary. In other more benign cases it can corrupt data integrity and lead to ineffective application of other data analysis techniques. To tackle the problem of anomaly detection, there are several common methods provided in statistics and machine learning literature, including variational autoencoder (VAEs). Using a VAE, we develop a novel objective function to improve its performance detecting anomalies. Additionally, we introduce a modeling pipeline that works in the fully unsupervised context, where one does not know the true proportion of anomalies present in the data. To construct this pipeline, we fit reconstruction errors using a Gaussian mixture model (GMM) and select the model whose characteristics best match our performance metrics. Using our approach, we observe an increase in anomalies detected against a standard objective function, and we measure an average improvement of 0.4021 in F1 scores. We show our findings using four labeled benchmark data sets and apply our conclusions on an open-source, unlabeled data set taken from USASpending.gov.
Document Details
- Document Type
- Technical Report
- Publication Date
- Jun 01, 2019
- Accession Number
- AD1080012
Entities
People
- Brandon Bowman
Organizations
- Naval Postgraduate School