Flow-Induced Mathematical Theory for Deep Learning: Bridging the Gap between Theory and Practice
Abstract
Overview. Deep learning essentially regards a simple question: what is the theoretical and real-world power of the class of function,s described by matrix-vector products interwoven with nonlinear functions when trained by stochastic optimization methods? From an a,pplication standpoint, deep neural networks (DNN) have proven highly successful at solving a variety of empirical machine learning p,roblems. Yet, this success is poorly understood from a theoretical standpoint matching the actual practice in applications. This pro,ject seeks to fill the gap between theory and practice by introducing new flow-induced mathematical theories. More specifically, thi,s project treats practically applied optimization algorithms as the theoretical hub of deep learning theory and clarifies: (i) the a,pproximation power of DNNs that is achievable via optimization; (ii) the generalization error of DNNs implicitly induced by optimiza,tion; (iii) the stability and perturbation of DNNs determined by optimization trajectories. The work proposed will lay the foundatio,n for an understanding of the properties of DNNs (including both their structure and implementation), the properties of an optimizat,ion algorithm (including existing and novel algorithms proposed here), and the properties of an estimation problem through generaliz,ation analysis. From these foundations, the hope is to bridge the gap between deep learning theory and practice to make deep learnin,g more reliable and trustworthy for real applications.Intellectual Merit. The proposed research will result in an understanding of t,he fundamental mechanisms underlying the success and limitations of deep learning as well as theoretical guarantees and justificatio,ns to support empirical results. The anticipated outcome is a comprehensive theory encompassing the practical, mathematical, and sta,tistical aspects of deep learning, including new mathematical characterizations of deep neural networks induced by optimization grad,ient flows that yield insights into their approximation capabilities, convergence properties, and prediction uncertainty, and new me,thods of data analysis built on mathematical and statistical theory to enable the successful use of deep neural networks on challeng,ing learning problems.Broader Impacts. The recent developments in machine learning have been called a revolution by countless author,s. Many see the tremendous potential of using such techniques in areas that directly impact human life and military applications, su,ch as healthcare, automation, human-computer interaction, ocean and climate prediction, decision making and long-term planning, amon,g many others. Others have questioned whether such enthusiasm is justified. Can learned models truly replace human expert insights?,Are there dangers to using modeling tools that we do not completely understand? Might there be some unintended consequences to putti,ng so much trust in models derived solely from data? The theory and methods developed in this project will help to provide answers t,o these critical questions.This project summary is approved for public release.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Apr 01, 2022
- Source ID
- N000142212341
Entities
People
- Haizhao Yang
Organizations
- Office of Naval Research
- Purdue University
- United States Navy