What Is Your Metric Telling You? Evaluating Classifier Calibration under Context Specific Definitions of Reliability

Abstract

Definitions of ECE that focus solely on the predicted class fail to accurately measure calibration error under a selection of practically useful definitions of reliability. Many common calibration techniques fail to improve calibration performance uniformly across ECE metrics derived from these diverse definitions of reliability.

Open PDF

Document Details

Document Type
Technical Report
Publication Date
May 11, 2022
Accession Number
AD1168397

Entities

People

  • Eric Heim
  • Jacob Oaks
  • John Kirchenbauer

Organizations

  • Carnegie Mellon University

Tags

Communities of Interest

  • C4I

DTIC Thesaurus Topics

  • Calibration
  • Copyrights
  • Department Of Defense
  • Engineering
  • Governments
  • Guarantees
  • Intervals
  • Intervention
  • Lepidoptera
  • Machine Learning
  • Materials
  • Reliability
  • Software Development
  • Test And Evaluation
  • Universities

Readers

  • Aerospace Test and Evaluation
  • Systems Analysis and Design