Note on Learning Rate Schedules for Stochastic Optimization,

Abstract

We present and compare learning rate schedules for stochastic gradient descent, a general algorithm which includes LMS, on-line back-propagation and k-means clustering as special cases. We introduce search-then-converge type schedules which outperform the classical constant and running average (l/t) schedules both in speed of convergence and quality of solution.

Document Details

Document Type
Technical Report
Publication Date
Jan 01, 1992
Accession Number
ADP007159

Entities

People

  • Christian J. Darken
  • John Moody

Organizations

  • Yale University

Tags

DTIC Thesaurus Topics

  • Algorithms
  • Clustering
  • Computer Science
  • Convergence
  • Data Science
  • Engineering
  • Evolutionary Algorithms
  • Heuristic Methods
  • Information Science
  • Learning
  • Mathematics
  • Optimization
  • Statistics
  • Theoretical Computer Science

Fields of Study

  • Computer science

Readers

  • Adaptive Control and Estimation with Uncertainty in Dynamic Systems.
  • Brain and Cognitive Science; Experimental Psychology; Cognitive Neuroscience
  • Neural Network Machine Learning.