Using Hybrid Methods for Relevance Assessment in TREC Crowd'12

Abstract

The University of Iowa (UIowaS) submitted three runs to the TRAT subtask of the 2012 TREC Crowdsourcing track. The task objective was to evaluate approaches to crowdsourcing high quality relevance judgments for a text document collection. We used this as an opportunity to examine three hybrid (combination of human-based and machine-based) approaches while simultaneously limiting time and cost. We create a training set from topics, which were previously assessed for relevance on the same document set, and use this training set to build strategies. We apply machine approaches, including clustering, to order documents for each topic, and then ask crowdworkers to provide relevance judgments for a subset of documents. One of our runs provides the best logistic average misclassification (LAM) rates of all submitted TRAT runs.

Open PDF

Document Details

Document Type
Technical Report
Publication Date
Nov 01, 2012
Accession Number
ADA581318

Entities

People

  • Christopher Harris
  • Padmini Srinivasan

Organizations

  • University of Iowa

Tags

Communities of Interest

  • Materials and Manufacturing Processes

DTIC Thesaurus Topics

  • Abstracts
  • Clustering
  • Coefficients
  • Crowdsourcing
  • Information Operations
  • Judgment
  • Simulations
  • Standards
  • Statistical Samples
  • Statistical Sampling
  • Test Sets
  • Training
  • United States
  • Universities

Fields of Study

  • Computer science

Readers

  • Information Retrieval
  • Instructional Design and Training Evaluation.
  • Neural Network Machine Learning.