Solving Complex Tasks with Team-Based Crowdsourcing
Abstract
Short Work StatementWe will pursue systems and science for team-based crowdsourcing: crowd teams can address far more complex and interdependent work than traditional crowdsourcing techniques, while capitalizing on the crowd s diversity of expertise and ability to dynamically assemble and re-assemble itself to match any goal. Specifically, we will create systems enabling teams of expert workers to dynamically assemble from the crowd that are 1) matched to the task at hand, and 2) effective collaborators.ObjectiveToday s crowdsourcing systems ask a large number of non-experts to make independent contributions that can be merged to produce expert-level results. While these crowdsourcing approaches are effective for routine and modular tasks, they struggle with many real-world goals such as software development, design, and problem solving. Our objective is to draw together on-demand teams of experts from the crowd that can accomplish such problems, which involve interdependencies that are difficult to decompose into independent microtasks. First, we will develop team matching systems that gather on-demand crowd members who perform well together and are available simultaneously. Second, we will design peer assessment algorithms and social structures that produce calibrated reputation signals forcrowd members.ApproachWe achieve the objective through two projects that address core problems in team-based crowdsourcing. The first project focuses on team member matching, or convening members of the crowd who are likely to perform well together. The second project focuses on peer assessment, or developing crowd worker guilds that can establish credible reputation signals in a crowdsourced environment where teams and hierarchy are fluid. The core technical contributions of this research will be 1) a team matching algorithm that identifies members based on expertise, familiarity and personality, all subject to recruitment time deadlines, and 2) a calibrated peer assessment algorithm that can estimate a worker~s ability in a domain of expertise.ONR Mission/RelevanceOn-the-ground naval operations require rapid assembly of teams to interpret incoming information, stage a plan, and execute. Since these teams may rotate members rapidly and alter their makeup based on task needs, these teams need carefully selected members who maximize the collective ability to achieve desired outcomes. In such scenarios, systems and methods such as those described in this proposal could play a transformative role. Our systems could help such teams identify which of potentially available members should be recruited, and even help perform the recruitment and establish team rooms for each effort. In addition, these rapidly-assembled naval teams will need to develop trustable distributed reputation signals. Again, systems and methods in this proposal could enable such teamsto quickly establish trust through credible signals. Even if these naval teams assemble and disassemble for each new task, these methods would allow the organization to use peer assessment to retain some memory of who is most effective.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Sep 30, 2016
- Source ID
- N000141612894
Entities
People
- Michael Bernstein
Organizations
- Office of Naval Research
- Stanford University
- United States Navy