Storytelling with Receiver Operating Characteristic (ROC) Curves for Environmental Remediation of Unexploded Ordnance (UXO)
Abstract
Receiver operating characteristic (ROC) curves are often used to assess the performance of binary classification systems, allowing stakeholders to understand tradeoffs between Type I (false positive) and Type II (false negative) errors. This works well in textbook cases. In real-world experiments, however, ROC curves can have unexpected subtleties that make them difficult to construct and interpret. For example, the Department of Defense is sponsoring the development of novel sensors and software to identify unexploded ordnance (UXO) in the midst of clutter. UXO are duds--munitions that were previously armed and fired but failed to explode. UXO can still pose a risk of detonation even decades later, threatening the safety of nearby humans, animals, vegetation, and structures. Conducting blind tests to demonstrate finding UXO is fraught with safety, logistical, and cost constraints that make it difficult to construct the textbook ROC curves. Yet, with careful planning and a few key assumptions, ROC-like curves can still be crafted to quickly tell the story of how well novel sensors and software can detect, classify, and locate UXO versus clutter in terrestrial and underwater experiments.
Document Details
- Document Type
- Technical Report
- Publication Date
- Jul 01, 2021
- Accession Number
- AD1199469
Entities
People
- Jacob B. Bartel
- Shelley M. Cazares
Organizations
- Institute for Defense Analyses