Interactive Spatial Object Search Engine
Abstract
One of the key capabilities of an intelligent robot is to search and find different objects in large, cluttered environments. The aim of this proposal is to test the hypothesis that interactive decision-theoretic agents with 4D spatiotemporal visual perception (4D = 3D + time) and dialog capabilities can efficiently find a wide range of objects in large-scale environments. This is hard since environments may be only partially observed, object descriptions may be incomplete (e.g., any screwdriver), and search can fail in cluttered scenes with occlusions. Previous work in visual object search treats this as a passive object detection problem with both the environment and target objects fully observed and described. They also lack higher-level reasoning through human–robot dialog to recover from search failures. In this proposal, we will conduct fundamental research to address these limitations by building an interactive spatial object search engine. It consists of decision-theoretic object-based agents with 4D spatiotemporal perception capabilities enabling operation in partially-observed environments. To handle incomplete target object descriptions, our engine will incorporate deep learning models for learning conditioned category-level object models – models that are informed by both category-level priors (e.g., all screwdrivers), and sparse conditioning through natural language (e.g., red screwdriver) or images. Our 4D perception and object modeling capabilities can detect failures but not recover from them. To address this, our search engine will support dialog-based human–robot collaboration informed by 4D perception and object modeling (e.g., red screwdriver behind a magazine in the dining room). Together, our interactive decision-theoretic agents can inform the movement and manipulation actions of robots equipped with our engine. We will extensively evaluate and demonstrate our approach both at the component and system-wide levels by assessing search success rate, speed, generalization, and efficiency of human–robot teams.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Mar 07, 2023
- Source ID
- FA95502110214
Entities
People
- Srinath Sridhar
Organizations
- Air Force Office of Scientific Research
- Brown University
- Office of the Secretary of Defense