Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundations

Abstract

We develop the mathematical foundations of the stochastic modified equations (SME) framework for analyzing the dynamics of stochastic gradient algorithms, where the latter is approximated by a class of stochastic differential equations with small noise parameters. We prove that this approximation can be understood mathematically as an weak approximation, which leads to a number of precise and useful results on the approximations of stochastic gradient descent (SGD), momentum SGD and stochastic Nesterovs accelerated gradient method in the general setting of stochastic objectives. We also demonstrate through explicit calculations that this continuous-time approach can uncover importantanalytical insights into the stochastic gradient algorithms under consideration that may not be easy to obtain in a purely discrete-time setting.

Open PDF

Document Details

Document Type
Technical Report
Publication Date
Mar 01, 2019
Accession Number
AD1105415

Entities

People

  • Cheng Tai
  • Qianxiao Li
  • Weinan E

Tags

Communities of Interest

  • C4I
  • Materials and Manufacturing Processes

DTIC Thesaurus Topics

  • Algorithms
  • Artificial Intelligence
  • Computations
  • Difference Equations
  • Differential Equations
  • Eigenvalues
  • Equations
  • Information Processing
  • Information Systems
  • Machine Learning
  • Numerical Analysis
  • Partial Differential Equations
  • Probability
  • Random Variables
  • Stochastic Processes
  • Theorems
  • Two Dimensional

Fields of Study

  • Computer science

Readers

  • Control Systems Engineering.
  • Finite Element Method (FEM) for solving Partial Differential Equations (PDEs)
  • Systems Analysis and Design