Northeastern University Runs at the TREC12 Crowdsourcing Track
Abstract
The goal of the TREC 2012 Crowdsourcing Track was to evaluate approaches to crowdsourcing high quality relevance judgments for images and text documents. This paper describes our submission to the Text Relevance Assessing Task. We explored three different approaches for obtaining relevance judgments. Our first two approaches are based on collecting a limited number of preference judgments from Amazon Mechanical Turk workers. These preferences are then extended to relevance judgments through the use of expectation maximization and the Elo rating system. Our third approach is based on our Nugget-based evaluation paradigm.
Document Details
- Document Type
- Technical Report
- Publication Date
- Feb 06, 2013
- Accession Number
- ADA581299
Entities
People
- Javed A. Aslam
- Jesse Anderton
- Jie Wu
- Maryam Bashir
- Matthew Ekstrand-abueg
- Peter B. Golbus
- Virgil Pavlu
Organizations
- Northeastern University