Perceptive Whole-body Legged Manipulation in Unstructured Natural Environments
Abstract
Compliant humanoid legged manipulators have great potential to maneuver safely in tight spaces and perform rich types of tasks in en,vironments such as Navy ships, military bases, medical facilities and industrial setups. Albeit the tremendous progress that has bee,n made in building reliable and low-cost hardware for legged manipulation, they are primarily restricted to highly structured labora,tories and short term deployment. Real-world deployment leads to a series of fundamental technical challenges: The robots have to pe,rceive the unstructured scenes and make autonomous decisions in the presence of the variability and uncertainty in realistic environ,ments. Recent advances in robot perception, learning, and control have given,deep learning, reinforcement learning (RL), learning from demonstration (LfD), and trajectory optimization (TO), for designing intel,ligent robot behaviors. How- ever, to employ these techniques for legged manipulation, we need to address three intimately connected, research questions: 1) how will robots perceive the environments into abstractions that capture necessary information for simultane,ous locomotion and manipulation? 2) how will robots reason about and fulfill complex tasks involving both mobility and manipulation,, and generalize to novel tasks and new goals? 3) how will high-DoF robots robustly plan and learn their feasible motions while adapt,ing to the changing environment? This project aims to address these questions and develop abstractions, algorithms, and evaluations,for enabling intelligent and robust behaviors of legged manipulators in unstructured natural environments. Our research will lever,age the compliant Draco 3 humanoid robot recently funded by ONR to develop an integrated solution that endows legged manipulation ro,bots their ability to understand the unstructured world through raw visual perception, make decisions for long-time horizon tasks, a,nd execute the actions with robust trajectory optimization methods. At the core of our approach is a structured object-centric abstr,action of an unstructured environment, called hierarchical scene graphs. This abstraction captures the geometric properties of the o,bjects, their functional properties in the form of affordance maps, and their pairwise semantic relations. With this abstraction, ou,r research will devise: a) active perception algorithms to estimate scene graph abstractions from robots raw sensory input; b) meth,ods to use these abstractions to learn from human demonstrations to perform neurosymbolic planning in long-time horizon tasks; and c,) robust trajectory optimization methods that integrate data-driven models and versatile trajectory generation capabilities to maneu,ver and manipulate objects in the complex environments using object-centric frames. This research will accelerate the deployment o,f effective and reliable mobile legged manipulators in naval vessels and in industrial setups. It will lead to transformative applic,ations of robotic technologies in various types of tasks, including maintenance and repair, digital twin scanning, firefighting, ins,pection and short distance logistics. Methods and principles developed in this re- search will reduce the technical barriers of depl,n to military advantages this research includes outreach activities through UT Austins Freshman Research Initiative (FRI), mentorin,g female graduate students, recruiting undergraduate researchers from under-represented groups for research, and dissemination of re,search outcomes through open-source initiatives and curriculum development. Approved for public release
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- May 16, 2022
- Source ID
- N000142212204
Entities
People
- Luis Sentis
Organizations
- Office of Naval Research
- United States Navy
- University of Texas at Austin