Self-Optimizing & Deliberative Workflows
Abstract
Crowdsourcing - solving a problem through the computer-coordinated efforts of myriad humans - has become extremely popular with the" growth of labor markets, such as Mechanical Turk and citizen science platforms. Recentwork has demonstrated that workflows can be"" used to solve problems as complex as document translation, videoconstruction, and software development. However, quality control of" such tasks continues to be a key challengebecause of the high variability in worker quality. Most existing quality-control approac"hes rely on the aggregation of responses from multiple, independent workers, using techniques ranging from simple, majority vote to"" methods basedon expectation maximization. While these methods yield accurate results in some cases, they often converge to local m"axima and hence fail on difficult problems where a majority of workers get the answer wrong. This has caused some researchers to con"clude that crowdsourcing can never achieve high accuracy, no matter how many workers are involved. We argue that a radically-differe"nt approach is necessary to achieve highly-accurate crowd processes.Instead of keeping workers in isolation and aggregating their o"utput with blind statistical measures, better results may be achieved with techniques for micro-collaboration and argumentation. Eve""n on hard problems, where most workerserr, a single worker may convince the majority to switch to a better outcome through a sound" argument. Now is the perfect time to investigate these methods because preliminary results show that argumentation can be extremely"successful. We have recently shown that contextual, multi-turn arguments offer dramatic improvements in quality (e.g., 89% reductio""n in error, controlling for cost). But much work is necessary to achieve the full potential of argumentationfor crowdsourcing. Mult""iTalk, while the best system today, has significant limitations: it is expensive, limited to binary decisions, and requires a carefu""lly-structured set of task guidelines. We propose to address these limitations with the following innovations: OPTIMIZED, MULTI-TUR""N ARGUMENTATION. There are several ways to reduce the labor (and associated expense) from multi-turn argumentation. First, argumenta"tion may not be necessary for all problems. We will devise decision theoretic & reinforcement-learning methods for deciding when and" how the system should invoke argumentation. Secondly, structured argumentation - where rules and guidelines can be quickly issued b"y clicking on guidelines and templates (rather than by typing natural language text) - should enable faster argument creation and re"duced cognitive load. Third, we will develop efficient, asynchronous argumentation methods. SELF-IMPROVING GUIDELINES. Guidelines (a""lso known as rubrics), which structure the terms and logic underlying a task, are essential to good decisions. But if a task is poor""ly understood and has incomplete annotation guidelines, then arguments may not be definitive. In this case the guidelines need to be" revised and we conjecture that the inconclusive argument structure can focus the process of guideline augmentation. We will investi"gate meta-workflows that engage the crowd to 1) propose guideline extensions which would resolve controversies, 2) summarize,restru""cture, distill and simplify guidelines, and 3) argue and vote about the best changes. Our research will have significant impact on f""uture naval decision making. As the complexity of missions grow larger, it is increasinglyimportant to make high-quality decisions"" quickly and accurately, even when participants are physically distributed and separated by time-zone barriers that preclude synchro"nous communication. We expect to deliver substantially more accurate results for any amount of labor assigned.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Mar 26, 2018
- Source ID
- N000141812193
Entities
People
- Daniel S. Weld
Organizations
- Office of Naval Research
- United States Navy
- University of Washington