Reducing Test Length with Polychotomous Scoring
Abstract
This study compared two new types of test scoring (polyweighting and dichotomous Item Response Theory (IRT) scoring) to traditional number-correct (NC) scoring. The study used data collected from applicants for military enlistment who had taken the Armed Services Vocational Aptitude Battery (ASVAB). The objective was to determine the impact of these two new scoring methods on alternate-form reliabilities. Content areas studied were ASVAB Mathematics Knowledge (MK) and General Science (GS). Examinees in one part of the study completed one of five experimental MK tests and one of six operational MK tests. Other examinees in the study completed one of four experimental GS tests and one of six operational MK tests. Other examinees in the study completed one of four experimental GS tests and one of six operational GS tests. Experimental and operational tests were analyzed separately. For each type of test, at each possible test length, median alternate-form reliability was determined for each scoring method. The authors reached five conclusions: (1) Polyweighting is superior to NC scoring; (2) dichotomous IRT scoring is usually superior to NC scoring, but not always; (3) when dichotomous IRT scoring works well, polyweighting and dichotomous IRT scoring allow similar reductions in test length; (4) polyweighting and IRT scoring both provide greater benefits when item difficulties are more variable; and (5) polyweighting and IRT scoring use different (complementary) mechanisms for increasing measurement precision
Document Details
- Document Type
- Technical Report
- Publication Date
- Oct 01, 1993
- Accession Number
- ADA277761
Entities
People
- J. B. Sympson
- Mark L. Davison