Crowdsourcing the Hardest Questions
Abstract
Crowdsourcing has emerged as a revolutionary approach for quickly solving problems that resist fully-algorithmic approaches and require human knowledge or judgment. Since individual crowd members are human, they can err, and a variety of quality-control algorithms have been developed to ensure accurate output. Unfortunately, most existing approaches rely on aggregation of multiple, independent answers in their attempt to ensure quality. While these methods yield accurate results in many cases, they fail on extremely difficult problems where the majority of workers get the answer wrong. This has caused many researchers to conclude that crowdsourcing can never achieve an accuracy of greater than 85-90% on some tasks, no matter how many workers are involved. In contrast, we believe that the fundamental problem stems from the simplistic aggregation approaches used to date. We propose four methods for improving the accuracy of crowdsourcing: microtalk, Bayesian crowd reflection, decision-theoretic question routing, and differential instruction. If successful, our methods will significantly improve the accuracy of both commercial and research crowdsourcing methods.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Aug 12, 2016
- Source ID
- N000141512774
Entities
People
- Daniel S. Weld
Organizations
- Office of Naval Research
- United States Navy
- University of Washington