Analogical World Models- Perception, Action and Language Grounding through Analogical Prediction
Abstract
The PI, Dr. Karerina Fragkiadaki, propose an analogical framework for knowledge representation, perception and action from images and videos that encodes domain knowledge explicitly, in a collection of structured sensory experiences at different levels of spatial and temporal abstraction, in addition to implicitly, as network parameters. The proposed model retrieves memories and uses them to modulate perceptual inference to localize or generate analogous entities in the sensory stream, detect objects, parts, attributes, action-events, generate possible future and past action and event completions, evaluate counterfactuals, ground referentials, answer questions, and act in the environment. Each memory experience is encoded as a spatial-temporal graph of perceptual entities alongside a symbol(s), the symbols for roles, attributes, objects and actions.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Feb 29, 2024
- Source ID
- FA95502310257
Entities
People
- Katerina Fragkiadaki
Organizations
- Air Force Office of Scientific Research
- Carnegie Mellon University
- United States Air Force