Features of the Geometry of Neural Networks
Abstract
Deep learning is driving the development of technologies of increasing power. At present, our ability to implement and train neuralnetworks outpaces our understanding of what takes place inside of them. However we are succeeding in steadily and methodically building this knowledge. Artificial neural networks learn through a training process that takes place on a loss landscape L defined fromthe neural network, via a gradient descent based algorithm. Therefore as we continue to probe how deep neural networks learn, an essential aspect is to improve our understanding of the geometry of the loss function L that underlies each deep neural network. Here,we propose to use tools from mathematics to build this understanding of this geometry that is so important to deep learning. Previous work of the PI and others has provided steady progress in elucidating the geometry of this loss landscape. New results from the PI and collaborators have opened the door to bringing tools from Morse theoryand Morse homology to study the geometry of L. In this proposal, we look to carry out that vision through a multi-year effort to adapt this powerful machinery to the setting of deep neuralnetworks. In doing so, we strive to shed light on a fundamental and long-standing question in the mathematical foundations of deep learning: to better understand the relative prevalence of different types of critical points. This is a fundamental question, which has proved challenging to address. We believe that by adapting tools from mathematics to the setting of deep neural networks, more powerful than those which have been used thus far, we will be able to shed new light on these previously studied questions. The geometry of the loss landscape is a fascinating subject, and we look with excitement to the next stage in building our knowledge of thesesystems.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Nov 09, 2024
- Source ID
- N000142412607
Entities
People
- Yaim Cooper
Organizations
- Office of Naval Research
- United States Navy