Understanding and implementing multi-scale neuro-glial dynamics for robust non-Markovian learning and decision making
Abstract
Reinforcement learning (RL) is a key framework to model and understand how neural circuits acquire complex functions as a result of time and experience. Perhaps as a consequence, RL has emerged as a powerful and commonly used paradigm in machine learning (ML) and artificial intelligence (AI). However, there remains a significant gap between the often slow, fragile and computationally-demanding performance of ML-RL systems, and the fast, robust and efficient learning exhibited by animals and humans. In particular, many ML-RL agents learn on a single time-scale wherein decisions are assumed to be Markovian in nature, meaning that they depend only on current circumstances without regard for prior actions. Such an assumption aids analytical and computational tractability but strongly restricts the class of problems that can be treated and leads to limits in scalability and generalizability of learning. In particular, a typical ML-RL agent tries to maximize reward based on its current state by regressing or updating its decision parameters against prediction errors. While intuitive in its simplicity, this scheme is slow to adapt when a task context changes, for example being asked to subtract after first learning addition. Conversely, humans and animals are non-Markovian actors, able to accrue and integrate their prior experiences over disparate time-scales and strategically seek information to reduce future uncertainty, even if doing so does not meet immediate task goals. As a result, human and animal agents are readily adaptable to changing environmental dynamics and task requirements and can generalize their past experiences towards the acquisition of new functions. The biological mechanisms that support this level of sophistication are highly enigmatic, which may explain the disconnect between biological and machine learning performance. We propose an integrative engineering-neuroscience approach that will close these gaps in knowledge and capability. The overall goal of this proposal is to construct new RL-ML methods that obviate existing challenges by validating a novel theory of neurobiological learning based on the dynamical interactions of neurons and astrocytes, a key type of glial cell. Despite their ubiquity in the brain, the functional role of glia is poorly understood. Our central hypothesis is that astrocytes and neurons form two levels of a coherent, reciprocally connected hierarchical network where astrocytes integrate contextual information and modulate neural circuitry in a manner that is favorable for large-scale learning. We postulate that neuronal and synaptic processes can be rapidly influenced by astrocytes, allowing learning to occur quickly when task circumstances change, and for prior contextual information to optimally constrain learning when new requirements are faced. To validate this idea and embed it into new algorithmic solutions, our proposal takes an expansive approach that combines expertise in engineering, machine learning, neuroscience and genetics. We will develop a first-of-its-kind theoretical framework that links neuro-glial dynamics to high-level learning functions through bottom-up and top-down modeling and analysis approaches. We will test predictions emanating from our theory through paradigm-shifting innovations in experiment design, where we enact causal manipulations of astrocytes in non-human primates, so as to be able to probe high-level, non-Markovian behavioral paradigms. The tight integration of theory and experiments will result in significant knowledge gain across engineering and neuroscience, culminating in new RL algorithms that meet the increasing demand for AI to learn and operate robustly in complex, uncertain, and dynamic decision environments.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Oct 07, 2021
- Source ID
- W911NF2110312
Entities
People
- ShiNung Ching
Organizations
- Army Contracting Command
- United States Army
- Washington University in St. Louis