Crowdsourcing assessments and evaluations
Abstract
It is often the case that the performance (or ability, experience, knowledge, productivity, etc.) of an agent(professionals, researchers, authors etc.) cannot be easily assessed by an objective ~test~ but rather must be inferred from the reports or evaluations of other agents. For instance, the administrator of a hospital cannot directly evaluate the performance of a nurse but must rely on the reports of other nurses, and the editor of a journal cannot fully evaluate the quality of an article but must rely on the reports of referees. In these and many other settings, it is necessary or convenient that the evaluation process be ~crowd sourced~. However, crowdsourcing systems for evaluations/assessments often do not operate well because ~peer~ evaluations may be unreliable for any of a number of reasons: the evaluator may be malicious or incompetent or may simply not exert sufficient effort to provide an accurate assessment because effort is not rewarded. To ensure that crowdsourcing systems for valuations/assessments work well it is necessary to identify high quality (accurate) feedback and to provide incentives for evaluators to provide such feedback, and in particular, to encourage effort and cooperative behavior anddiscourage low quality feedback due to limited effort spent, incompetence or malicious behavior. The object of this research is to develop formalisms, methods and an associated ~crowdsourcing~ platform foreffectively performing evaluations/assessments using other peers. The methods developed will be very widelyapplicable in many scenarios, including those mentioned above ~ indeed to almost all settings in which the quality of the work of an individual is subjective, or cannot be directly tested or easily observed/evaluated by the authority ~ but can be observed/evaluated by peers.This research will build on the PI~s long-standing expertise on repeated games, and design of rating and matching systems but will also involve developing new game-theoretic methods which go beyond the existing work in stochastic and repeated games with imperfect monitoring, social norms, rating and matching systems. Thus, the project will lead to the development of new game theoretic formalisms, methods and designs.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Feb 03, 2017
- Source ID
- N000141712215
Entities
People
- Mihaela Van Der Schaar
Organizations
- Office of Naval Research
- United States Navy
- University of Oxford