Evaluating Machine Learning Classifiers for Hybrid Network Intrusion Detection Systems
Abstract
Existing classifier evaluation methods do not fully capture the intended use of classifiers in hybrid intrusion detection systems (IDS), systems that employ machine learning alongside a signature-based IDS. This research challenges traditional classifier evaluation methods in favor of a value-focused evaluation method that incorporates evaluator-specific weights for classifier and prediction threshold selection. By allowing the evaluator to weight known and unknown threat detection by alert classification, classifier selection is optimized to evaluator values for this application. The proposed evaluation methods are applied to a Cyber Defense Exercise (CDX) dataset. Network data is processed to produce connection-level features, then labeled using packet-level alerts from a signature-based IDS. Seven machine learning algorithms are evaluated using traditional methods and the value-focused method. Comparing results demonstrates fallacies with traditional methods that do not consider evaluator values. Classifier selection fallacies are revealed in 2 of 5 notional weighting schemes and prediction threshold selection fallacies are revealed in 5 of 5 weighting schemes.
Document Details
- Document Type
- Technical Report
- Publication Date
- Mar 26, 2015
- Accession Number
- ADA622990
Entities
People
- Michael D. Rich
Organizations
- Air Force Institute of Technology