Sigmoidal Weight Constraint in a Recurrent Neural Network
Abstract
When training a recurrently connected neural network (RNN), the magnitude of the connection strengths (weights) must be limited in some way. The weights are normally constrained by either renormalizing them after each learning step, or by using a decay term proportional to the weight. For large numbers of training cycles, we show that an RNN output can become unstable with previously used weight adjustment methods. We introduce a technique that constrains weight values to move on a smooth sigmoidal curve. Without the need for renormalization or a parametric decay term, our RNNs then produce stable output. Performance is also improved in other ways. As an example, an associative memory RNN is shown to converge much faster and to more accurate values than with previous methods.
Document Details
- Document Type
- Technical Report
- Publication Date
- Dec 01, 2004
- Accession Number
- ADA436300
Entities
People
- Simon A. Barton