YIP - Holistic Scene Understanding Through Neuro-Symbolic Visual Representation Learning
Abstract
We propose to build intelligent systems that understand scenes holistically: from low-level, detailed object geometry, material, and reflectance, to high-level scene structure, regularity, and repetition, and deeply integrated with semantic concepts supplied by natural language. Such a system will be able to support a wide range of practical tasks in vision, robotics, and mixed reality, from shape reconstruction and novel view synthesis, to object recognition and interaction, to visual reasoning and question answering, torobotic manipulation and task planning. Recent advances in computer vision and AI are significant, but mostly domain- and task-specific, and their integration is lacking. We propose to integrate them with fundamentally new scene representations, Neuro-Symbolic Visual Representations (NSVRs), which model scenes across multiple, physically-based, and cognitively-inspired levels of abstraction. Our main idea is to integrate higher-level, structural, and often symbolic priors in the physical world, including the notion of objects, surfaces, parts, and their hierarchical structure, with lower-level neural representations for object intrinsics, such as their fine geometry, texture, reflectance, and physics. We further propose new algorithms that infer an NSVR from heterogeneous data sources, that leverage NSVRs for various vision, robotics, and multisensory tasks, and that integrate NSVRs with natural language for visual reasoning, question answering, and planning. The abstract is approved for public release.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Jan 24, 2024
- Source ID
- N000142412117
Entities
People
- Jiajun Wu
Organizations
- Office of Naval Research
- Stanford University
- United States Navy