Crowdsourcing Assessments and Evaluations
Abstract
Statement of Work:In many operation settings, the performance (or ability, experience, knowledge, productivity, etc.) of an agent(professionals, researchers, authors etc.) cannot be easily assessed by an objective ~test~ but rather must be inferred from the reports or evaluations of other agents. In such settings, it is necessary or convenient that the evaluation process be ~crowdsourced~. However, crowdsourcing systems for evaluations/assessments often do not operate well because ~peer~ evaluations may be unreliable for any of a number of reasons: the evaluator may be malicious or incompetent or may simply not exert sufficient effort to provide an accurate assessment because effort is not rewarded.To ensure that crowdsourcing systems for evaluations/assessments work well it is necessary to identify high quality (accurate) feedback and to provide incentives for evaluators to provide such feedback, and in particular, to encourage effort and cooperative behavior and discourage low quality feedback due to limited effort spent, incompetence ormalicious behavior. The goal of this project is to construct a system for crowdsourcing assessments that can operate in many different communities/types of agents in an efficient, fair and robust manner. The theory and the associated constructions willneed to enable us to deal with the incentive problems posed by assessments in a large variety of communities of professionals, including communities consisting of a large number of anonymous, heterogeneous (have different needs, preferences, capabilities, information etc.) agents, and in which the peers have heterogeneous ability and evaluations are (known to be) imperfect.Objective:The object of this research is to develop formalisms, methods and an associated ~crowdsourcing~ platform foreffectively performing evaluations/assessments using other peers. The methods developed will be very widelyapplicable in many scenarios, including those mentioned above ~ indeed to almost all settings in which the quality of the work of an individual is subjective, or cannot be directly tested or easily observed/evaluated by the authority ~ but can be observed/evaluated by peers.Approach:In this project the PI aims to develop a systematic framework for modeling the problem of crowdsourcing assessments, a theory for analyzing it, methods for solving it, algorithms for designing and implementing it and a prototype for validating it. Typically, evaluations require costly effort and do not convey direct benefit. This will mean that, in the absence of incentives, evaluators will not want to put in the effort needed to evaluate others. However, as mentioned previously, evaluators are in turn evaluated by others (not necessarily those that they evaluated), so social reciprocity can be used in the design of the crowdsourcing platform for assessments in such a way that if the evaluator does exert effort and provides an accurate evaluation, then he/she can be rewarded for this effort in the form of being assigned better evaluators in the future; this is desirable (at least for agents who produce good work) because better evaluatorsproduce more accurate assessments of quality and also because it is believed that the quality of an agent as a worker will improve if he/she receives high quality evaluations. A succinct way to keep track of the evaluator~s history of past assessments and their qualities is through a rating/reputation system. The rating will be used to reward/punish agentsthrough the assignment of better/worse sets of evaluators.Overall Merit and ONR Mission/Relevance:Providing high quality, competent and truthful assessments of the work performed by various agents - the ~agents~ may be individuals or teams of individuals ~ is of paramount importance for the well-functioning of any organization and/or operation, including the Navy and/or Naval operations. The systems developed in this project for crowdsourcing evaluations will be relevant to numerous Navy opera
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Aug 12, 2016
- Source ID
- N000141612660
Entities
People
- Mihaela Van Der Schaar
Organizations
- Office of Naval Research
- United States Navy
- University of California, Los Angeles