Grounded action transformation for sim-to-real reinforcement learning
Abstract
Reinforcement learning in simulation is a promising alternative to the prohibitive sample cost of reinforcement learning in the physical world. Unfortunately, policies learned in simulation often perform worse than hand-coded policies when applied on the target, physical system. Grounded simulation learning (gsl) is a general framework that promises to address this issue by altering the simulator to better match the real world (Farchy et al. 2013 in Proceedings of the 12th international conference on autonomous agents and multiagent systems (AAMAS)). This article introduces a new algorithm for gsl—Grounded Action Transformation (GAT)—and applies it to learning control policies for a humanoid robot. We evaluate our algorithm in controlled experiments where we show it to allow policies learned in simulation to transfer to the real world. We then apply our algorithm to learning a fast bipedal walk on a humanoid robot and demonstrate a 43.27% improvement in forward walk velocity compared to a state-of-the art hand-coded walk. This striking empirical success notwithstanding, further empirical analysis shows that gat may struggle when the real world has stochastic state transitions. To address this limitation we generalize gat to the stochasticgat (sgat) algorithm and empirically show that sgat leads to successful real world transfer in situations where gat may fail to find a good policy. Our results contribute to a deeper understanding of grounded simulation learning and demonstrate its effectiveness for applying reinforcement learning to learn robot control policies entirely in simulation.
Document Details
- Document Type
- Pub Defense Publication
- Publication Date
- May 13, 2021
- Source ID
- 10.1007/s10994-021-05982-z
Entities
People
- Garrett Warnell
- Haresh Karnan
- Josiah P. Hanna
- Peter Stone
- Siddharth Desai
Organizations
- Defense Advanced Research Projects Agency
- General Motors
- Lockheed Martin
- National Science Foundation
- Office of Naval Research
- Robert Bosch
- United States Army Research Laboratory