Improving Analytic Reasoning via Crowdsourcing and Structured Analytic Techniques
Abstract
How might analytic reasoning in intelligence reports be substantially improved? One conjecture is that this can be achieved through a combination of crowdsourcing and structured analytic techniques (SATs). To explore this conjecture, we developed a new crowdsourcing platform supporting groups in collaborative reasoning and intelligence report drafting using a novel SAT we call “Contending Analyses.” In this paper we present findings from a large study designed to assess whether groups of professional analysts working on the platform produce better-reasoned reports than those analysts produce when using methods and tools normally used in their organizations. Secondary questions were whether professional analysts working on the platform produce better reasoning than the general public working on the platform; and how usable the platform is. Our main finding is a large effect size (Cohen’s d = 1.37) in favor of working on platform. This provides early support for the general conjecture. We discuss limitations of our study, implications for intelligence organizations, and future directions for the work as a whole.
Document Details
- Document Type
- Pub Defense Publication
- Publication Date
- Aug 17, 2020
- Source ID
- 10.1177/1555343420926287
Entities
People
- Ariel Kruger
- Ashley Barnett
- Elizabeth Silver
- Glenn Tesla Jayaputera
- Mark Burgman
- Morgan Saletta
- Richard De Rozario
- Richard O. Sinnott
- Sujai Thomman
- Tim van Gelder
Organizations
- Imperial College London
- Intelligence Advanced Research Projects Activity
- University of Melbourne