Lifelong Multimodal Fusion by Cross Layer Distributed Optimization
Abstract
We propose a three year research project on the development of (multi-agent) cooperative lifelong learning theory and practice, so as to accomplish adversary-resilient, fast-adaptable, and communication- and computation-efficient multimodal information fusion systems in dynamic environments. The motivation for this project is to tackle a series of new challenges emerged in distributed intelligence, including lifelong learning (LL), multimodal representation learning, and adversarial robustness. To circumvent these challenges, we develop a principled and unified optimization framework, termed cross-layer distributed optimization (CLDO), in which networked agents collectively process the gathered data to solve a multi-level optimization problem formulated to simultaneously fulfill various learning requirements such as prediction accuracy, robustness to adversaries, and quick adaptation to new environments. Our proposal is organized by three intimately connected research thrusts: Thrust I seamlessly integrates lifelong learning with adversarial learning and meta learning to tackle the question of ?how to achieve lifelong, robustness-aware multimodal learning in a single-agent setup??. Thrust II addresses the question of ?what content should be shared between agents and how?? by developing a novel model sharing scheme built upon two pillars, (1) knowledge distillation from shared experience, and (2) lifelong model pruning that reduces communication costs without incurring catastrophic forgetting. Thrust III aims at acquiring computationally efficient machinery to solve CLDO problems using accelerated first-order optimization and Bayesian non-convex optimization. Through our proposal, we envision a future networked learning system that is resilient, adaptive, and scalable, which can seamlessly interact with machine learning techniques (especially with deep learning) as well as support distributed intelligence across numerous heterogeneous devices ? from parameter servers in the cloud to personal devices such as smartphones, and to intelligent sensors and other dedicated computing edges. The success of our proposal will be validated via two representative use cases, (1) distributed class-incremental visual recognition, and (2) personalized federated learning for heterogeneous agents, via extensive simulation, modeling, testing, and prototyping efforts. We believe that our proposal could make a significant impact on a variety of army missions, e.g., situation awareness, edge computing, federated learning, and air/ground reconnaissance. We intend to collaborate with Dr. Lance Kaplan and Dr. Brian Sadler at the Army Research Laboratory to maximize translational impact of our work.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Sep 01, 2023
- Source ID
- W911NF2310343
Entities
People
- Sijia Liu
Organizations
- Army Contracting Command
- Michigan State University
- United States Army