Deep Learning of Sensor-Driven Prehensile and Non-Prehensile Manipulation
Abstract
This project will be decomposed into three tasks, each of which will investigatea different class of approaches to the multimodal end-to-end robotic manipulation problem. Task 1: Prediction. In this task, we will study how we can train deep predictive models that predict future sensor observations for multiple sensory modalities. We will study how multiple modalities can support one another to produce more accurate and longer-term predictions, and how thesepredictive models can be utilized to enable a robot to perform tasks in the real world. Task 2: Reinforcement Learning. In this task, we will study reinforcement learning algorithms for end-to-end control, focusing on methods that utilize multi-modal perception and transfer experience from prior tasks to enable more effective exploration for new tasks. Task 3: Imitation Learning. We will study imitation learning from the standpoint of multi-modal embedding, examining how deep probabilistic models can jointly represent multiple sensory modalities,enabling a robot to imitate demonstrations in one modality (e.g., text, or videos of humans) by using its own sensory modalities (e.g., onboard cameras and touch sensors). Anticipated Outcome. The proposed research will advance the state of the art in robotic learning, enabling autonomous acquisition of sensorimotor skills that generalize more widely due to powerful, generalizable, learned representations.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Jan 23, 2019
- Source ID
- N000141912042
Entities
People
- Sergey Levine
Organizations
- Office of Naval Research
- United States Navy
- University of California Regents