Supporting Human Learning in the Context of AI-Assisted Decision-Making
Abstract
Approved for Public ReleaseModern AI-powered decision support tools directly support decision-makers by providing decision recommendations, risk scores or various forms of outcome predictions. This contrasts with many of the previous generation of decision-support aids, which aided decision makers by helping them make sense of and transform information. The modern decision support tools were developed with the hope that human operators would naturally integrate their own knowledge with that surfaced by the machines and, together, would produce better decisions than either people or machines could on their own. We now understand that achieving such human-AI complementarity is hard and there is plenty of recent work systematically exploring numerous mitigating strategies and documenting what ways of structuring the human-AI collaboration lead to best decisions for what kinds of problems and settings.While this work is happening in the research community, AI-powered decision support tools are being deployed in a growing number of real-world settings to a growing number of human decision-makers. The goal of this proposal is to address two urgent, but fundamental challengesrelated to the large-scale deployment of modern AI-powered decision support tools in real-world settings:1.Deskilling the workforce. Unlike decision support offered by human coworkers, support from AI tools in the form of a decision recommendation and explanationdoes not result in incidental learning about the problem domain.2.Lack of clear guidelines for what human-AI interaction approachesare the most effective for which settings and objectives. The design knowledge is growing but it is currently fragmented with insufficient theoretical foundations to offer clear guidance regarding the generalizability of any new empirical finding.To address thesechallenges, we will pursue the following objectives:1.Systematically evaluate existing and novel human-AI interaction approaches with respect to their ability to support human learning. The key intellectual contributions of Objective 1 will be an extensive and systematic characterization of how different human-AI interaction approaches, including some novel paradigms that do not involve presenting decision-makers with decision recommendations, impact decision-makers learning about the domain.2.Support deployment of AI-powered decision support tools by developing and evaluating an offline reinforcement learning approach for creating dynamic human-AI interaction policies for new settings. The key intellectual contribution of Objective 2 will be a validated and reusable method for rapidly developing an effective human-AI interaction strategy for a novel real-world setting; a method that can be applied in the absence of clear and generalizable theoretical guidelines.3.Support generalizability of empirical knowledge related to the design of AI-powered decision support tools by developing an RL-based method for rapidly evaluating which aspects of the problem space matter. The key intellectual contributions of Objective 3 are twofold: First, we will develop a novel method that will enable fast (compared to existing methods) and rigorous investigation of what aspects of the decision task, people making the decisions, and the context matter to the generalizability of design knowledge. Second, we will characterize several of these dimensions (individually and in combination with others).
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Nov 09, 2024
- Source ID
- N000142412726
Entities
People
- Krzysztof Gajos
Organizations
- Office of Naval Research
- President and Fellows of Harvard College
- United States Navy