An Evaluation of Polyweighting in Domain-Referenced Testing
Abstract
This technical note describes an empirical evaluation of a polychotomous item scoring procedure developed by the first author. This new scoring procedure (polyweighting) assigns an empirically-derived scoring weight to each possible response to a test question. An examinee's polyscore is equal to the mean of the scoring weights of the response categories chosen by the examinee. In this research, polyweighting was applied to test data obtained from 1, 1 00 resident physicians who had completed a 200-item medical certification test. Using the 200 items as an item bank, the authors assembled 20 short (10-, 20-, 30-, 40-item) assessment tests and used both proportion-correct scores and polyscores from these short tests to predict each physician's score on the 200- item certification test. For all 20 assessment tests, polyweighting resulted in higher cross-validated internal-consistency reliability (coefficient-CL) and domain validity. The observed increases in reliability corresponded to a mean increase in test length of 28%. Over all 20 tests, the mean increase in domain validity was .075. The minimum increase in domain validity was .052.
Document Details
- Document Type
- Technical Report
- Publication Date
- Oct 01, 1993
- Accession Number
- ADA272833
Entities
People
- J. B. Sympson
- Thomas M. Haladyna