Network Event Correlation Using Unsupervised Machine Learning Algorithms

Abstract

We have successfully implemented a two-stage event correlation model for intrusion detection system (IDS) alerts. The model is designed to automate alert and incidents management and reduce the workload on an IDS analyst. We achieve this correlation by clustering similar alerts together, thus allowing the analyst to only look at a few clusters instead of hundreds or thousands of alerts. The first stage of this model uses an artificial neural network (ANN)-based autoassociator. The autoassociator is trained to reproduce each alert at its output, and it uses the error metric between its input and output to cluster similar alerts together. The accuracy of the system is improved by adding another machine-learning stage which attempts to combine closely related clusters produced by the first stage into super-clusters. The second stage uses the Expectation Maximisation (EM) clustering algorithm. The model and performance of this model are tested with intrusion alerts generated by a Snort IDS on DARPA's 1999 IDS evaluation data as well as incidents.org alerts.

Open PDF

Document Details

Document Type
Technical Report
Publication Date
Nov 01, 2006
Accession Number
ADA462898

Entities

People

  • Maxwell Dondo
  • Nathalie Japkowicz
  • Peter Mason
  • Reuben Smith

Organizations

  • Defence Research and Development Canada

Tags

Communities of Interest

  • Autonomy
  • Cyber
  • Materials and Manufacturing Processes

DTIC Thesaurus Topics

  • Algorithms
  • Data Mining
  • Data Sets
  • Detection
  • Detectors
  • Dimensionality Reduction
  • Information Science
  • Information Systems
  • Intrusion Detection
  • Intrusion Detection Systems
  • Intrusion Detectors
  • Machine Learning
  • Network Protocols
  • Neural Networks
  • Port Scanners
  • Self Organizing Systems
  • Unsupervised Machine Learning

Fields of Study

  • Computer science

Readers

  • Regression Analysis.
  • Sensor Fusion and Tracking Systems.

Technology Areas

  • AI & ML
  • AI & ML - Neural Networks