Spatiotemporal Interaction Capture Platform for Intelligent Agents
Abstract
[Approved for Public Release]For effective operation, intelligent agents#both humans and robots#need to understand how to interact with their surroundings: to search for, grasp, move, re-arrange, and manipulate scene objects. These physical interactions take place over different spatial and temporal (4D) scales, ranging from short tabletop interactions, to long search operations outdoors. Humans and other animals excel at these interactions, but despite advances, we lack an understanding of the sensorimotor capabilities needed to build robots that have similar abilities. This is partly due to limitations in computer vision-based perception of 4D spatio-temporal changes, in particular the lack of strong priors and a repository of knowledge about interactions. Multi-camera capture systems and neural fields for shape and appearance capture have matured enough to now be used for the problem of creating these priors and repository of interaction skills. We refer to this problem as interaction capture#the capture of spatio-temporal changes in shape, appearance, motion, contact, and high-level actions that constitute interactions of both humans and robots with their surroundings. In this DURIP proposal, we propose to acquire the instrumentation capability to capture high fidelity audiovisual data of intelligent agents#both humans and robots#interacting with their surroundings. Specifically, we propose to design and assemble a multi-purpose Spatio-Temporal Interaction Capture (STIC) platform to capture rich, high-fidelity, multi-scale audiovisual interaction data. STIC will consist of two cuboidal aluminum structures assembled from off-the-shelf components and fitted with components including hundreds of cameras, tens of depth sensors, microphones and controllable LEDs. We will use these sensors to capture high fidelity audiovisual data about interactions that take place within the capturevolume of the structures. We will capture diverse human/robot interactions, for instance, human bodies, hands interacting with objects, robot arms manipulating objects, and mobile manipulators searching for and picking objects. We also propose to acquire storage and computate hardware for processing this data and creating a large repository of interaction skills. To our knowledge, STIC will be the largest interaction capture platform of its kind. STIC will enhance the research and educational activities of three ongoing DoD projects, and two ongoing NSF projects ranging from interactiveobject search to dexterous teleoperation. The proposed work will develop technology for both military and civil application.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Aug 11, 2023
- Source ID
- N000142312804
Entities
People
- Srinath Sridhar
Organizations
- Brown University
- Office of Naval Research
- United States Navy