Brain-inspired deep learning models of visual reasoning
Abstract
The robust and efficient recognition of visual relations in images is a hallmark of human vision. Despite recent progresses in visual recognition, modern machine vision algorithms (DCNs or Deep Convolutional Networks) are severely limited in their ability to learn visual relations, which restricts their large-scale application for most military and security uses.Based on the PI~s previous work and preliminary studies, we suggest that these fundamental limitations are related to the fact that current models rely primarily or exclusively on feedforward processing. Such processing is sufficient for certain visual recognition tasks including the recognition of objects presented in isolation or in the presence of limited background clutter and distractor objects. However, such processing does not allow for more complex visual reasoningtasks that are believed to require additional feedback routines including mental transformations, perceptual grouping, attentional selection and working memory. Much work is currently being devoted to extending DCNs to include such mechanisms but these models are still outmatched by the power and versatility of the brain, perhaps in part due to the richer neuronal computationsavailable to cortical circuits. The challenge is to identify which neural mechanisms are relevant, and to find suitable abstractions to model them.The assumption underlying this research is that a concerted effort to understand and emulate the information-processing strategies used in biological systems will enable the development of ground-breaking DoD technologies. The proposed research will identify the perceptual principles and model the brain mechanisms underlying visual reasoning with the following specific aims:Aim 1 will develop a novel visual reasoning challenge used to identify the minimal set of necessary computations by systematically evaluating modern DCN architectures representing an array of computational strategies (including pure feedforward, spatial and feature-based attention, state-dependent memory as well as various perceptual grouping and binding mechanisms.) Rather than dealing with more realistic but elaborate spiking neuron models, Aim2 concerns the development of novel machine learning idealizations of key neural operations, which are trainable with machine learning algorithms but still interpretable at the cellular level.Together this project will yield a substantial advance in our scientific understanding of visual cognition via the development of rigorous computational models that span multiple levels of analysis from circuits to system, that are able to predict human behavioral data during visual reasoning tasks, and that surpass the capabilities of current state-of-the-art computer vision systems. Progress in our understanding of the mechanisms underlying vision, combined with innovations in computer graphics, imaging, machine learning, and high-end computing hardware makes this research topic viable and timely. This project is interdisciplinary in nature and should have broad impact in multiple disciplines including human perception, neuromorphic engineering and computer vision. Understanding which neural computations are carried by our visual cortex would give scientists a powerful tool to uncover key mechanisms of humanperception and cognition as well as to create a new generation of cognitive architectures with broad military applications ~ from improving human-based visual analysis to alleviating its limitations via artificial vision systems and smart assistive devices.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Jan 23, 2019
- Source ID
- N000141912029
Entities
People
- Thomas Serre
Organizations
- Brown University
- Office of Naval Research
- United States Navy