Multi-Agent Communication Learning in Discrete Channels with Unknown Noise
Abstract
This work is inspired by the common observation that when agents, each of limited capability,work together in groups, complex communication behaviors spontaneously emerge, enablingagents to perform complex tasks as a single coordinated agent. It seems as if new capabilitiesemerge from nothing. We seek to create a new simple-to-complex approach to multi-agentcommunications that embraces and then advances the state of the art in machine learningfor multi-agent control. Already, great researchers, such as those at DeepMind and OpenAI,have produced state-of-the-art multi-agent systems that learn automatically to consistentlyoutperform humans at three-dimensional strategy-intensive tasks, albeit almost exclusivelyin simulated games like StarCraft, Dota 2, and virtual capture the flag [8, 11, 16], withlimited application to physical robotic systems. Our goal is to develop techniques for efficientcommunications learning in the context of multi-agent reinforcement learning (MARL) onrealistic robotic systems.While our interest lies in endowing real robotic systems with improved coordinationcapability, these results in simulated games still highlight an important issue: agents donot directly share information, and instead learn fully decentralized policies. Particularly inpartially-observable environments, where agents do not have sufficient information from theirown sensor measurements alone, such a lack of information sharing among agents results insuboptimal policies and possibly poor performance at the group level. Endowing agents withthe ability to selectively exchange information using a communication network allows themto supply each other with the information required to make moreinformed local decisions,ultimately allowing superior global performance. However, this additional capability posesadditional challenges, such as how to select the best information to send, how to encode itinto a message, and how to interpret messages from other agents.A core challenge, to be addressed by the proposed work, is the issue of communicationthrough discrete communications. This isimportant because in reality, communicationsnetworks are almost always discrete, whereas much prior work assumes they are continuous.Essentially, we seek to overcome this challenge by enabling gradient backpropagation (whichcurrently is only possible in continuous processes) to work through discrete channels. To thurethat allows discrete channels to be modeled as analog channels with additive noise.This effort will incorporate an encryption-based message-passing technique that enablesunbiased gradient estimation in the presence of unknown channel noise. We will also exploreways in which these techniques for differentiating through realistic communication channelscan be combined with gradient-based learning approaches, such as model-free RL andimitation learning, to facilitate efficient communications learning in realistic environments.In addition to the learning algorithm, the agent architecture is also an important variablein ultimate performance. For this reason, we will evaluate the applicability of advancedneural network architectures, such as graph neural networks, which are uniquely suited tothe problem of MARL with communications. Finally, to further improve robustness tochannel noise, we will show that learning-based signal processing techniques, such as learnedchannel coding, can be naturally incorporated into our communications-learning framework.We will show that, because our approach is fully learned, it can leverage its adaptability toadversarial noise conditions to outperform classical channel coding techniques.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- May 05, 2021
- Source ID
- N000142112406
Entities
People
- Howie Choset
Organizations
- Carnegie Mellon University
- Office of Naval Research
- United States Navy