Estimating Interrater Reliability in Incomplete Designs.

Abstract

Estimates of interrater reliability are often needed for incomplete designs in which raters (e.g., employees) are nested within targets (e.g., organizations). It is shown that the popular use of estimates based on between-group ANOVAs accompanied by intraclass correlations can be seriously misleading if low variation exists among target means. An alternative based on a within-group procedure is proposed and shown to be superior to the intraclass correlation in the condition of low variation among group means, accompanied by low within-group variation. (Author)

Open PDF

Document Details

Document Type
Technical Report
Publication Date
Aug 31, 1981
Accession Number
ADA104041

Entities

People

  • Gerrit Wolf
  • Lawrence R. James
  • Robert G. Demaree

Organizations

  • Texas Christian University

Tags

DTIC Thesaurus Topics

  • Agreements
  • Aircrafts
  • Airplanes
  • Analysis Of Variance
  • Behavioral Research
  • Classification
  • Coefficients
  • Composite Materials
  • Data Science
  • Equations
  • Experimental Design
  • Frequency
  • Homogeneity
  • Information Science
  • New York
  • Random Variables
  • Reliability

Readers

  • Instructional Design and Training Evaluation.
  • Regression Analysis.