Assessing and Expanding the Limits of Collective Intelligence
Abstract
Due to several major advances in electronic communications and the growth of Web 2.0 technologies, Collective Intelligence has quickly transformed from a predominantly socio-theoretical concept into a widely deployed practical tool set for tactical decision-making, business, and various other domains. Along this transformation, Collective Intelligence methodologies have assumed various forms, but all share the common goal of aggregating or fusing multiple sources of potentially conflicting information into a whole that is greater than the sum of its parts. One of the most powerful variants of these methodologies is the concept of the Wisdom of the Crowd, which holds that when a collection of estimates is appropriately elicited and aggregated, the resulting estimate can be more accurate than most individual estimates, including those of subject-matter experts. Owing to the widely documented successful applications of crowd wisdom (e.g., prediction markets, prediction polls, solving combinatorial problems), a compelling line of inquiry is whether and how the classic conditions of this social phenomenon can be supplemented to achieve improved outcomes that can rival the accuracy of machine-based estimation approaches. This research project seeks to provide a proof of concept of novel methodologies for further teasing out crowd wisdom. This thrust is being pursued in order to augment human capabilities to solve problems that are computationally difficult and/or for which machine-based approaches are unsuitable. Research tasks will seek to construct judgment-elicitation and aggregation mechanisms that achieve higher accuracy in estimation tasks with objective ground truths. To this end, this project will develop computational tools that are capable of aggregating and gleaning insights from continuous, ordinal, and multimodal judgement-elicitation data. Optimization models and algorithms will be equipped to solve problems with large numbers of evaluation entities and agents efficiently. To test the efficacy of these tools, customized statistical models will be developed for generating nontrivial problem instances with objectively defined degrees of collective similarity/dissimilarity from an underlying ground-truth set. These will incorporate a variety of characteristics relevant to complex estimation tasks: incompleteness, sparsity, noise/error, imperfect subtask overlap, and high-dimensionality. The efficacy of the proposed aggregation methodologies will be quantified via empirical parametric descriptions of the relationships of varying elicitation choices and statistical distribution configurations with the similarity between the aggregate estimate and the underlying ground truth. The proposed tools may also be validated in practice through the implementation of crowd based estimation tasks. These would be guided by standard crowd wisdom benchmarks of varying difficulties such as ranking the difficulties of puzzles, counting randomly distributed dots in images, and box-office prediction. Tests may also be performed on relevant benchmarks from other applications (e.g., wireless sensor network estimation).
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Apr 22, 2019
- Source ID
- W911NF1910260
Entities
People
- Adolfo Escobedo
Organizations
- Arizona State University
- Army Contracting Command
- United States Army