Estimating Classifier Accuracy Using Noisy Expert Labels
Abstract
In this work, we present an empirical comparison of statistical methods that estimate the accuracy of a classifier using noisy expert labels. We are motivated by the application of machine learning to difficult problems for which even experts may be unable to provide an authoritative label for every data instance. Several estimators have been recently proposed in the literature, but prior empirical work to evaluate the applicability of these estimators to real-world problems is limited. We apply the estimators to labels simulated from three models of the expert labeling process and also four real datasets labeled by human experts. Our simulations reveal the importance of the accuracy of the classifier relative to the experts and confirm that conditional dependence between experts negatively impacts estimator performance. On two of the real datasets, the estimators clearly outperformed the baseline majority vote estimator, supporting their use in applications. We also briefly examine the utility, in terms of increasing or decreasing confidence in an estimators output, of a few diagnostics that can be applied to the expert labels.
Document Details
- Document Type
- Technical Report
- Publication Date
- Jan 31, 2018
- Accession Number
- AD1054207
Entities
People
- J. T. Holodnak
- J. T. Matterer
- W. W. Streilein
Organizations
- Massachusetts Institute of Technology