Learning Multi-Step Dexterous Bimanual Fine Manipulation for Shipboard Maintenance and Urban Exploration
Abstract
Approved for Public Release Our proposal focuses on addressing the challenges of multi-step dexterous bimanual fine manipulation. This problem is incredibly relevant, to our everyday lives and to marine operations. However, this problem is especially challenging because such tasks are contact-rich, demand high levels of force and positional feedback, and demand very precise tolerances, often at the limit of the manipulator s capabilities. Our approach is guided by the observation that the knowledge required for fine manipulation can be broken down into two distinct pieces, each requiring a different approach. First, commonsense-based knowledge by which we mean knowledge that is common across a wide variety of tasks, some not even related to fine manipulation. This means there isabundant data available and pretrained language models such as Large Language Models (LLMs) and image segmentation and classification models trained on internet data can be used zero-shot (with no finetuning) or perhaps very limited amount of task-specific finetuning. We plan to leverage these pretrained Foundation models significantly in our work. Second, expert knowledge by which we mean knowledge that is extremely specific to the fine manipulation task being accomplished. Here we make a key observation, namely that most interesting tasks are highly expert-data starved. There just isn#t a lot of expert data of bomb defusal or heart surgery or even the physical maintenance of infrastructure. This is the key reason why we cannot just take off-the-shelf models and apply them directly for fine manipulation. We plan to address data-scarcity in two fundamental ways. First, by leveraging mathematical structure of the problem or the dynamics to derive data-efficient learning methods. Second, by building simulators that can help the robot to experiment and learn structure by itself efficiently without compromising the safety of the environmentor itself. The focus application of our proposal is the physical maintenance of marine infrastructure. There are many shipboard tasks that are repetitive but critical for the safe and efficient operation of the vessel. Key among them are tasks that require the physical maintenance of infrastructure on the vessel, for example the maintenance and servicing of marine diesel engines. Although it is one among many such shipboard tasks, servicing marine diesel engines is a capstone example of the types of tasks we would like to automate via robots with the dexterity to perform fine manipulation. We propose to facilitate the automation of marine shipboard maintenance with robots via the following three thrusts: T1 Algorithms: Although the problem is incredibly challenging there is structure in the task that we can exploit that can help us enable robots to perceive, act, and learn fine manipula tion skills that can solve forceful bimanual fine manipulation tasks. T2 Demonstration on a real physical marine diesel engine and a standard testbed: We will use an existing highly-capable bimanual manipulator at the PI#s lab to demonstrate maintenance tasks in the real world in real-time. Inspired by the modular andinstrumented task boards for manipulation fromNASA and from NIST, we also propose to build an open-source 3D printed instrumented marine diesel engine to provide a standard testbed for robotics research. T3 Open-source simulation framework and assets: Not all academic labs will have access to a capable bimanual manipulator or the space to house a marine diesel engine. We propose to build a simulation environment capable of benchmarking real physical interactive tasks that need to be performed on the real engine. We will build a photorealistic environment for rendering and sensor simulation, as well as physically-realistic interaction. Additionally, wewill build a set of challenge problems using this simulator that will be released open-source.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Nov 08, 2024
- Source ID
- N000142412674
Entities
People
- Siddhartha Srinivasa
Organizations
- Office of Naval Research
- United States Navy
- University of Washington