SiLK: A Tool Suite for Unsampled Network Flow Analysis at Scale

Abstract

A large organization can generate over ten billion network flow records per day, a high-velocity data source. Finding useful, security-related anomalies in this volume of data is challenging. Most large network flow tools sample the data to make the problem manageable, but sampling unacceptably reduces the fidelity of analytic conclusions. In this paper we discuss SiLK, a tool suite created to analyze this high-volume data source without sampling. SiLK implementation and architectural design are optimized to manage this Big Data problem. SiLK provides not just network flow capture and analysis, but also includes tools to analyze large sets and dictionaries that frequently relate to network flow data, incorporating higher-variety data sources. These tools integrate disparate data sources with SiLK analysis.

Open PDF

Document Details

Document Type
Technical Report
Publication Date
Jun 01, 2014
Accession Number
ADA610679

Entities

People

  • Jonathan Spring
  • Katherine Prevost
  • Leigh Metcalf
  • Mark Thomas
  • Paul Krystosek

Organizations

  • Carnegie Mellon University

Tags

Communities of Interest

  • Autonomy
  • Cyber
  • Sensors

DTIC Thesaurus Topics

  • Algorithms
  • Big Data
  • Computer Network Security
  • Computer Networks
  • Computer Programs
  • Data Analysis
  • Data Storage Systems
  • Detectors
  • Engineering
  • Internet
  • Network Architecture
  • Network Protocols
  • Networks
  • Reliability
  • Sensor Networks
  • Software Development
  • Wireless Sensor Networks

Fields of Study

  • Computer science

Readers

  • Computer Programming and Software Development.
  • Distributed Systems and Data Platform Development