Overparameterized neural networks implement associative memory
Abstract
Development of computational models of memory is a subject of long-standing interest at the intersection of machine learning and neuroscience. Our main finding is that overparameterized neural networks trained using standard optimization methods provide a simple mechanism for implementing associative memory. Remarkably, this mechanism allows for the storage and retrieval of sequences of examples. This finding also sheds light on inductive biases in overparameterized networks: while there are many functions that can achieve zero training loss in the overparameterized regime, our result shows that increasing depth and width in neural networks leads to maps that are more contractive around training examples, thereby allowing for storage and retrieval of more training examples.
Document Details
- Document Type
- Pub Defense Publication
- Publication Date
- Oct 16, 2020
- Source ID
- 10.1073/pnas.2005013117
Entities
People
- Adityanarayanan Radhakrishnan
- Caroline Uhler
- Mikhail A. Belkin
Organizations
- Massachusetts Institute of Technology
- National Science Foundation
- Nvidia
- Office of Naval Research
- Ohio State University
- Simons Foundation