The Threat Detection System that Cried Wolf: Reconciling Developers with Operators

Abstract

The Department of Defense and Department of Homeland Security use many threat detection systems, such as air cargo screeners and counter-improvised-explosive-device systems. Threat detection systems that perform well during testing are not always well received by the system operators, however. Some systems may frequently cry wolf, generating false alarms when true threats are not present. As a result, operators lose faith in the systemsignoring them or even turning them off and taking the chance that a true threat will not appear. This article reviews statistical concepts to reconcile the performance metrics that summarize a developers view of a system during testing with the metrics that describe an operators view of the system during real-world missions. Program managers can still make use of systems that cry wolf by arranging them into a tiered system that, overall, exhibits better performance than each individual system alone.

Open PDF

Document Details

Document Type
Technical Report
Publication Date
Jan 01, 2017
Accession Number
AD1026995

Entities

People

  • Shelley M. Cazares

Organizations

  • Institute for Defense Analyses

Tags

Communities of Interest

  • Air Platforms
  • Biomedical
  • Counter IED
  • Human Systems

DTIC Thesaurus Topics

  • Acquisition
  • Air Traffic Control Systems
  • Classification
  • Control Systems
  • Department Of Defense
  • Department Of Homeland Security
  • Detection
  • Explosives
  • Explosives Detection
  • False Alarms
  • Homeland Security
  • Improvised Explosive Devices
  • National Security
  • New York
  • Security
  • Transportation
  • United States

Fields of Study

  • Computer science

Readers

  • Instructional Design and Training Evaluation.
  • Military History of the United States in the 20th Century.
  • Sensor Fusion and Tracking Systems.