Enhancing the Validity of Rating-Based Tests

Abstract

Profile similarity metrics (PSMs) can be computed for rating-based judgment tests, personality scales, and biodata inventories to supplement conventional measures and enhance scale validity. These metrics quantify: shape, the correlation between a respondents rating profile and the scoring key; scatter, respondent tendency to use more or less of the available rating scale; elevation, respondent tendency to systematically provide high or low ratings; and delta, respondent tendency to provide high or low ratings relative to the key. Analyses conducted for three projects confirmed theoretical expectations that PSMs can be used to accurately model distance score variance and increment the validity of distance scores against performance outcomes. Project 1 utilized three judgment tests and demonstrated that shape and delta metrics predicted supervisor performance ratings (R = .33), while elevation and shape metrics predicted career intent (R = .25). Project 2 utilized conventional personality scales and showed that PSMs provided incremental validity beyond distance scores against performance outcomes and documented the stability of the validity gains using an independent cross sample. Project 3 evaluated the use of PSMs to score experimental 9-point personality in addition to conventional 5 point personality scales. Project 3 analyses demonstrated that PSMs provided incremental validity against performance outcomes beyond distance scoring for the combined personality battery (R = .54 vs. R = .47). The third project also documented construct validity between overlapping constructs for the 5-point and 9-point scales. These results redefine validity expectations for personality/judgment constructs and demonstrate the efficacy of PSMs procedures to broaden the scope of psychological domains for which accurate measurement is possible. The U.S. Army Research Institute for the Behavioral and Social Sciences (ARI) supported this research project.

Open PDF

Document Details

Document Type
Technical Report
Publication Date
Dec 01, 2018
Accession Number
AD1067951

Entities

People

  • Alisha M. Ness
  • Amanda J. Koch
  • Peter J. Legree
  • Robert N. Kilcullen

Organizations

  • U.S. Army Research Institute for the Behavioral and Social Sciences

Tags

Communities of Interest

  • Cyber
  • Human Systems

DTIC Thesaurus Topics

  • Applied Psychology
  • Data Analysis
  • Descriptive Analytics
  • Elevation
  • Management Personnel
  • Measurement
  • Military Research
  • Personality
  • Personnel Selection
  • Psychological Phenomena And Processes
  • Psychology
  • Regression Analysis
  • Reserve Officer Training Corps
  • Social Psychology
  • Social Sciences
  • Statistics
  • Supervisors

Fields of Study

  • Psychology

Readers

  • Computational Modeling and Simulation
  • Organizational Psychology.
  • Psychometric Testing or Psychological Assessment.