Exploring Interactive Relevance Feedback With a Two-Pass Study Design
Abstract
Interactive query refinement is widely believed to improve the effectiveness of ranked retrieval but it can be difficult to leverage existing batch evaluation frameworks to quantify the relative benefits of alternative interaction designs. This paper uses the new two-pass interaction design of the Text Retrieval Conference's High Accuracy Retrieval from Documents (HARD) track to explore the design space for cluster-based interactive relevance feedback. Two sites contributed two techniques for cluster formation and three techniques for cluster labeling. The effectiveness of each technique was compared with lower bounds based on blind relevance feedback, and with upper bounds found with oracle-based techniques. The clustering techniques were found to yield potential benefits, but the automatically constructed cluster labels were found not to support sufficiently accurate cluster selection. Elicitation of a desired cluster descriptor was found to significantly improve the effectiveness of a subsequent retrieval pass. These results indicate that the affordable two-pass study design used in the HARD track can yield useful insights to guide future design decisions.
Document Details
- Document Type
- Technical Report
- Publication Date
- Oct 01, 2004
- Accession Number
- ADA453566
Entities
People
- Daqing He
- Dina Demner-fushman
- Douglas W. Oard
Organizations
- University of Maryland