Optimization for Distributed Machine Learning
Abstract
Major Goals: The major goals of this project consist of the following aspect. (a) Design communication-efficient algorithms over general complex network to enable decentralize machine learning. (b) Design asynchronous and randomized methods over a star network to enable federated learning. (c) Design efficient algorithms to enable robust learning. Army relevance: to learn from data collected by agents (e.g., airfighters, autonomous vehicles) distributed over networks. To achieve these goals, we pursued the following approaches. (a) Propose algorithms with optimal communication complexity for decentralize/federated learning. (b) Propose algorithms with graph topology independent gradient and sampling complexity. (c) Propose novel algorithms for solving minmax and general variational inequalities from robust learning. The Scientific Barriers Exist in the lack of understanding about the interaction among communication of agents, computation performed by agents, data (samples) collected by agents, and the impact to solution accuracy. Removing these barriers will enable the design of efficient algorithms for learning over networks in terms of all three aspects, i.e., communication, computation and sampling. Accomplishments: Our accomplishments are listed as follows. (a) Developed novel decentralized communication sliding methods that can judiciously skip communication. (b) Developed graph topology independent method for decentralized and stochastic optimization. (c) Developed accelerated stochastic algorithm for nonconvex distributed finite-sum and multiblock optimization. (d) Developed new optimal methods for robust optimization and machine learning. The major conclusions include: (a) Communication complexity for stochastic optimization can be as small as for deterministic problems. (b) Gradient and sampling complexity can be independent of graph topology.
Document Details
- Document Type
- Technical Report
- Publication Date
- May 14, 2021
- Accession Number
- AD1224090
Entities
People
- George Lan
Organizations
- Georgia Tech Research Corporation