Representation-based Reinforcement Learning for Autonomous Systems
Abstract
Approved for Public ReleaseIn the rapidly growing field of autonomous systems, especially within defense applications, the paramount challenge lies in navigating complex, uncertain, and dynamic environments with systems that can learn and adapt in real-time. DeepReinforcement Learning, leveraging the versatility and adaptivity of deep neural networks, has become a powerful tool for addressing these challenges. However, deep learning models, though very flexible, introduce significant challenges for both learning and control due to their inherent complexities, highlighting the need for innovative representation-based approaches; in particular, system representation in abstract forms that are versatile enough to encapsulate a variety of dynamical systems, yet streamlined to enhancethe efficiency of learning and control planning processes. Our proposal introduces a novel approach centered on representation-based reinforcement learning to optimize the performance of such dynamical systems. Our method aims to bridge the gap between simulated environments and real-world scenarios, enabling the development of robust and adaptable control strategies.Central to our proposed method is a unified representation framework that represents the system dynamics and its value function in a linear form with respectto nonlinear feature spaces. This not only facilitates efficient dynamic programming or linear programming approaches for optimal policy seeking, but also enables a thorough analysis of performance guarantees, including optimality, adaptivity, robustness, and safety for autonomous systems. Our technical approach involves three research thrusts: i) data-driven representation learning for both offline and online settings, which addresses the challenge of learning from interactions within dynamic environments to build the foundations for the proposed approaches; ii) robust and safe learning, which aims to develop robust and safe control and learning algorithms that account for the inherent uncertainties and constraints in dynamic models learned from data; and iii) multi-task and multi-agent transfer learning, which exploit the representation to perform the transfer of skills among tasks and agents, enhancing the efficiency and adaptability of autonomous systems. All developed algorithms will be validated across simulation benchmarks and real-world robotic platforms. These thrusts collectively enable the developed methods to achieve modeling flexibility, robustness, and safety guarantees across different tasks and agents, and thus offer a comprehensive strategy that combines theoretical rigor with computational efficiency.Anticipated outcomes of this research include the development of advanced reinforcement learning algorithms that exhibit unprecedented levels of flexibility, robustness, adaptability, and efficiency with theoretical guarantees. These algorithms are expected to significantly enhance the defense capabilities in deploying autonomous systemsacross a wide range of operational scenarios, from reconnaissance and surveillance to complex mission planning and execution in contested environments, ensuring that autonomous systems can operate efficiently and safely in complex, dynamic, and uncertain environments.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Mar 12, 2025
- Source ID
- N000142512173
Entities
People
- Na Li
Organizations
- Office of Naval Research
- President and Fellows of Harvard College
- United States Navy