Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T
Abstract
The model‐free algorithms of “reinforcement learning” (RL) have gained clout across disciplines, but so too have model‐based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This “generalized reinforcement learning” (GRL) model, a frugal extension of RL, parsimoniously retains the single reward‐prediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversal‐learning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With high‐resolution high‐field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value‐based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.
Document Details
- Document Type
- Pub Defense Publication
- Publication Date
- Jul 21, 2022
- Source ID
- 10.1002/hbm.25988
Entities
People
- Arthur W. Toga
- Camilla van Geen
- Catherine A. Hartley
- Danielle Bassett
- Daphna Shohamy
- Harang Ju
- Jaron Colas
- John P. O’Doherty
- Joshua I Gold
- Julian Michael Tyszka
- Karol P Szymula
- Koranis Tanwisuth
- Natalie M. Saragosa‐Harris
- Neil M. Dundon
- Raphael T Gerraty
- Scott T. Grafton
Organizations
- Army Research Office
- California Institute of Technology
- Columbia University
- National Institute for Mathematical and Biological Synthesis
- National Institute of Biomedical Imaging and Bioengineering
- National Institute of Mental Health
- National Institute on Drug Abuse
- New York University
- University of California, Santa Barbara
- University of Freiburg
- University of Pennsylvania
- University of Southern California