Statistical Models for Criterion-Referenced Testing and Decisionmaking
Abstract
The purposes of this study were to investigate the characteristics of a well-constructed criterion-referenced performance test and to compare several statistical models that might help interpret criterion-referenced test scores. The models were compared on the accuracy of the pass or fail decisions which they implied and the accuracy of their estimates of examinee true scores. The three models that were considered share the binomial probability distribution for describing the expected distribution of observed scores given an examinee's true ability, and all define ability on a scale from 0 to 1.0. The first model, the proportion correct model, uses the proportion of responses that are correct as its estimate of true ability. Pass or fail criteria are set by considering the probabilities that examinees of differing abilities will achieve a variety of proportion correct scores. The score that would be expected to produce the least amount of classification error is chosen as the criterion score. The second model, the binomial error model, uses the observed score distribution to compute the regression of true score on observed score. Pass or fail decisions and true score estimates are based on the results of applying the regression equation. The third model, the beta-binomial Bayesian model, uses prior beliefs of expert judges to establish a prior ability distribution. Observed data are combined with the prior distribution to produce a posterior ability distribution for each observed score. Pass or fail decisions and true score estimates are based on the posterior distributions.
Document Details
- Document Type
- Technical Report
- Publication Date
- Oct 01, 1979
- Accession Number
- ADA080651
Entities
People
- Kenneth I. Epstein
Organizations
- U.S. Army Research Institute for the Behavioral and Social Sciences