DURIP Exploring the Local Geometry of Deep Networks
Abstract
ONR Program Officer: Behzad Kamgarparsi, Code 311, Machine Learning, Reasoning, and Intelligence--Publicly Releasable--Deep learning has significantly advanced our ability to address a wide range of difficult signal processing, machine learning, and artificial intelligence (AI) problems. Today s machine learning landscape is dominated by deep (neural) networks (DNs), which are constructed by piecing together a huge number of simple, local operations. Despite the simplicity of the local operations, there exists no clear and simple global representation for the operation of the entire network, including what goes on inside its many layers. Consequently,the field remains shrouded in mystery, with a lack of clarity regarding the "nobs" that can be tuned to design a high performing system. In promising preliminary research, we have discovered an elegant connection between a broad class of classical and modern DNsand continuous piecewise-affine (CPA) splines. Splines hold great potentialfor the geometric visualization, analysis, and interpretation of a DN in terms of how it partitions its input space during learning. However, exact computation of the spline partition has been precluded due to its combinatorial complexity.In promising preliminary research supported by ONR MURI grant N00014-20-1-2787, "Mathematical Foundations of Deep Learning," we have developed SplineCam, a computationally efficient method to compute localized slices of the exact spline partition for a wide class of DNs. SplineCam enables scalable new geometric ways and means to explore and characterize the inner workings of industrial-scale DNs. It has broad applicability for visualization, interpretation, and analysis across the entire design/build/test methodology of deep learning practitioners.However, access to powerful computing hardware is the bottleneck to SplineCam research, which motivates this DURIP instrumentation request. While SplineCam is the first tool capable of exactly exploring thelocal geometry of large-scale DNs, its quadratic computational complexity means that SplineCam research studies require large-scale, GPU-based computation capabilities in order to repeatedly train and make inferences from networks with hundreds of millions of parameters using massive training data sets. The theoretical and practical outcomes of this project have the potential to alleviate many of the troubling issues the DOD faces with DNs - including lack of interpretability and lack of robustness - and pave the way for more principled improvements to deep learning architectures that have predictable performance.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Mar 15, 2024
- Source ID
- N000142412225
Entities
People
- Richard G. Baraniuk
Organizations
- Office of Naval Research
- Rice University
- United States Navy