Practical Applications of Bloom Filters to the NIST RDS and Hard Drive Triage

Abstract

Much effort has been expended in recent years to create large sets of hash codes from known files. Distributing these sets has become more difficult as these sets grow larger. Meanwhile the value of these sets for eliminating the need to analyze "known goods" has decreased as hard drives have dramatically increased in storage capacity. This paper evaluates the use of Bloom filters (BFs) to distribute the National Software Reference Library's (NSRL) Reference Data Set (RDS) version 2.19, with 13 million SHA-1 hashes. We present an open source reference BF implementation and validate it against a large collection of disk images. We discuss the tuning of the filters, discuss how they can be used to enable new forensic functionality, and present a novel attack against bloom filters.

Open PDF

Document Details

Document Type
Technical Report
Publication Date
Sep 01, 2008
Accession Number
ADA486533

Entities

People

  • Douglas White
  • Paul Farrell
  • Simson Garfinkel

Organizations

  • Naval Postgraduate School

Tags

Communities of Interest

  • Biomedical

DTIC Thesaurus Topics

  • Application Software
  • Collisions
  • Computer Programming
  • Computer Programs
  • Computers
  • Cybersecurity
  • Data Sets
  • Databases
  • Hash Tables
  • Metadata
  • Microarchitecture
  • Operating Systems
  • Standards
  • Trees (Data Structures)
  • Virtual Machines

Fields of Study

  • Computer science

Readers

  • Cybersecurity.
  • Database Systems and Applications
  • Systems Analysis and Design