Measuring Classifier Intelligence
Abstract
Classifiers are seen here as systems in which input feature values are used with fitted or learned functions that produce output values which are interpreted as probabilities or fuzzy degrees of class membership, or in which output values are used with cut-off decision rules to choose bivalent class membership. Two complementary measurements for evaluating training, validation, testing, and deployment phase performances in human, mechanical, and computerized classifiers are proposed here. These measurements are derived from samples of classifier output values paired with their corresponding known probabilistic, fuzzy, or bivalent classification values. The first measurement is the area under the ROC plot. The second is the separation index newly introduced here. Both of these measurements are easy to understand and to compute. It is proposed that they be considered standard metrics for evaluating and comparing classifier intelligence.
Document Details
- Document Type
- Technical Report
- Publication Date
- Aug 01, 2002
- Accession Number
- ADA511068
Entities
People
- Jim Deleo
Organizations
- National Institutes of Health