A Theory-Based Concept Learning Framework for Perception, Reasoning, and Planning
Abstract
Flexible, efficient, and generalizable machine intelligence requires identifying and using compositional, grounded concepts. When making pour over coffee, humans first grind coffee beans to get grounded coffee, and then boil and pour water on it. Humans also associate these concepts (objects, attributes, events, relations, and actions) through an integrated loop of perception, reasoning, and planning- they extract them from language, ground them on visual input, and connect them via causal task planning. Existing machine intelligence systems, especially those that are purely based on statistical or deep learning, however, fall short in both aspects- they are mostly not interpretable or generalizable to new environments, and they often focus on solving a specific type of problems, such as image recognition. Inspired by the theory-theory of human concept learning, we propose to develop a computational representation of concepts, as well as methods that infer them from and use them for tasks in visual perception, language understanding and reasoning, and task planning. Our key insight is to represent concepts in a theory-based, neuro-symbolic way. The symbolic structure will be our focus and the framing skeleton of the concepts, representing the constituents of the theory, including its visual grounding, lexicon entry, and pre-conditions and post-conditions for planning. Deep learning will be used as minor components to capture features not easily characterized symbolically, such as the semantic meaning of a word. Our preliminary studies have indicated that such representation and algorithms enable data-efficient learning of visual concepts as well as combinatorial generalization based on these learned concepts on tasks across multiple domains. We hope the proposed research activities will further push the boundary for generalizable machine intelligence, with broad applications in perception, reasoning, and planning.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Feb 29, 2024
- Source ID
- FA95502310127
Entities
People
- Jiajun Wu
Organizations
- Air Force Office of Scientific Research
- Stanford University
- United States Air Force