Continual Learning with Differentiable External Memory of Orthogonal Representation for Previous Knowledge
Abstract
Continual learning is one of the hot topics in machine learning to overcome the problem of catastrophic forgetting. Conventional approaches in machine learning constructs a model from a given dataset, which has inherent inadequacy to learn incrementally over time for a series of data from new tasks, and it is impractical to retrain a model over all the previous datasets. To cope with this problem, we attempt to devise a novel method of continual learning that incorporates differentiable external memory of orthogonal representation for previous knowledge obtained from a sequence of tasks.This method exploits the orthogonal representation to more effectively store and retrieve previously learned knowledge, reducing interference between tasks and improving generalization capability. The differentiable memory allows us to efficiently manipulate previously knowledge in a differentiable form, resulting in efficient transfer of the stored knowledge in forward and backward directions. This might be a new breakthrough in continual learning because it can store the previous learned knowledge in a continuous distribution instead of discrete samples.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Feb 16, 2024
- Source ID
- FA23862314057
Entities
People
- Sung-Bae Cho
Organizations
- Air Force Office of Scientific Research
- United States Air Force
- Yonsei University