Brain-inspired deep learning models of visual reasoning
Abstract
Project Abstract [Approved for Public Release]Research problem Innovations in deep neural networks (DNNs) over the past decade havepowered many revolutionary applications, from chatbots that can sensibly respond to natural language queries, to image categorization and segmentation models that rival # and at times exceed # human accuracy. These impressive advancements have been powered by neural architectures that implement a scale-up of a processing mode that is akin to rapid feedforward processing in primates. Multiple lines of evidence have converged on the limitations of feedforward processing for solving tasks that require reasoning about objectsand their relationships. In comparison, a growing body of evidence suggests that many of the most challenging visual reasoning problems require feedback mechanisms. It is therefore not entirely surprising that even modern DNNs remain outmatched by the power and versatility of brains when solving abstract visual reasoning tasks. For example, the foundational transformer architecture is still unable to solve the #PathFinder# challenge accurately developed in the lab despite it being trivial for primate perception.Technical approachThere is an extraordinary opportunity for improving the reasoning capabilities of modern DNNs by incorporating feedback circuits that can perform similar computations as those in brains. In principle, such circuits should enable significant gains in generalization and learning efficiency by exploiting adaptive and reusable computational routines. However, to date, there is still relatively little known about the computational role of cortical feedback for reasoning about the visual world.Research objectiveThe objective of this research is to develop computational algorithms that explain how feedforward sensory processing interacts with computational feedback routines, including recurrent processes such as attention, semantics, short- and long-term memory, and mental simulation, to solve abstract reasoning tasks.Anticipated outcomeUnderstanding and emulating the feedback computations and associated neural dynamics of biological systems will enable the development of a radically new breed of vision architectures that exploit biologically-inspired algorithms for efficient and robust behavior. These models will also be extended to explain how feedback computations operate on modalities beyond vision, such as language and motor control, to enable more accurate, robust, and learning-efficient reasoning. These neuroscience-inspired DNNs will exhibit transformative capabilities, with the ability to operate as #digital twins# of human brains that can reliably predict behavior and generate testable mechanistic predictions on how they work. In contrast to the current scale-up of DNNs that has bolstered state-of-the-art performance, these neuroscience DNNs will rely on mathematical and computational principles, network topologies, and neural operations (including synchrony through complex valued representations, time delays, memory, attention, and other recurrent mechanisms) to achieve similar or better performance with greater sample complexity, while being implementable on compute-constrained devices.The proposed work is Fundamental Research and is to develop technology for both military and civil applications.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Nov 21, 2023
- Source ID
- N000142412026
Entities
People
- Thomas Serre
Organizations
- Brown University
- Office of Naval Research
- United States Navy