Extending Accelerated Optimization into the PDE Framework
Abstract
Much of the existing data mining literature is concerned with optimizing a finite set of parameters to train data classifiers. However, the sizes of these parameter sets are becoming increasingly large, not only in big data scenarios, but also in the context of training deep neural networks where the popular strategy of over- parameterization may increase the size of an initial parameter space by multiple orders of magnitude. Acceleration (also referred to as momentum) has become a crucial practical ingredient in most approaches for these massive optimization tasks. The investigator seeks to develop a new class of accelerated schemes in the infinite dimensional limit for distributed parameter spaces using partial differential equations (PDE). This will be a general framework designed to not only increase the speed of convergence but also greatly improve the robustness of gradient descent PDE-based optimization methods, particularly those connected with active contours, active surfaces, and level set methods in image processing and computer vision. The tools to be developed, will be general enough, however, to apply to region based motion estimation problems, including optical flow, as well as to problems in optical mass transport. The proposed research plan can therefore be succinctly stated as extending accelerated optimization methods into the PDE framework. Recent work has cast several popular acceleration schemes, heavily used in machine learning, into a variational ODE framework, yielding much more insight into the behavior and differences between existing schemes as well as clear principles around which new schemes can be developed. However, this existing knowledge is restricted to the finite dimensional setting of ordinary differential equations. To extend these same insights into the realm of partial differential equations, and in turn to leverage them for optimization in infinite dimensional (distributed parameter) spaces of curves, surfaces, and functions, several new mathematical, numerical, and computational considerations will have to be addressed that do not arise in finite dimensions. Accordingly, success in proposed effort, would yield broader and more generalized knowledge and understanding of acceleration methods in optimization. Keywords: Accelerated Optimization; Partial Differential Eqm1tions; Active Contours; Active Surfaces; Level Set Methods; Gradient Descent
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Feb 14, 2019
- Source ID
- W911NF1810281
Entities
People
- Anthony Yezzi
Organizations
- Army Contracting Command
- Georgia Tech Research Corporation
- United States Army