Modeling Temporal Crowd Work Quality with Limited Supervision

Abstract

While recent work has shown that a workers performancecan be more accurately modeled by temporal correlation intask performance, a fundamental challenge remains in theneed for expert gold labels to evaluate a workers performance.To solve this problem, we explore two methods of utilizinglimited gold labels, initial training and periodic updating.Furthermore, we present a novel way of learning a predictionmodel in the absence of gold labels with uncertainty awarelearning and soft-label updating. Our experiment witha real crowdsourcing dataset demonstrates that periodic updatingtends to show better performance than initial trainingwhen the number of gold labels are very limited (< 25).

Open PDF

Document Details

Document Type
Technical Report
Publication Date
Nov 11, 2015
Accession Number
AD1000128

Entities

People

  • Hyun J Jung
  • Matthew Lease

Organizations

  • University of Texas at Austin

Tags

Communities of Interest

  • Autonomy

DTIC Thesaurus Topics

  • Accuracy
  • Algorithms
  • Artificial Intelligence
  • Autocorrelation
  • Contracts
  • Crowdsourcing
  • Data Science
  • Information Science
  • Instructions
  • Intervals
  • Judgment
  • Learning
  • Machine Learning
  • Probability
  • Stationary Processes
  • Supervision
  • Test And Evaluation

Fields of Study

  • Computer science

Readers

  • Applied Combinatorial Optimization and Logic Circuit Design.
  • Computer Vision.
  • Organizational Process Management (OPM).