Nonlinear Information Bottleneck
Abstract
Information bottleneck (IB) is a technique for extracting information in one random variable X that is relevant for predicting another random variable Y. IB works by encoding X in a compressed “bottleneck” random variable M from which Y can be accurately decoded. However, finding the optimal bottleneck variable involves a difficult optimization problem, which until recently has been considered for only two limited cases: discrete X and Y with small state spaces, and continuous X and Y with a Gaussian joint distribution (in which case optimal encoding and decoding maps are linear). We propose a method for performing IB on arbitrarily-distributed discrete and/or continuous X and Y, while allowing for nonlinear encoding and decoding maps. Our approach relies on a novel non-parametric upper bound for mutual information. We describe how to implement our method using neural networks. We then show that it achieves better performance than the recently-proposed “variational IB” method on several real-world datasets.
Document Details
- Document Type
- Pub Defense Publication
- Publication Date
- Nov 30, 2019
- Source ID
- 10.3390/e21121181
Entities
People
- Artemy Kolchinsky
- Brendan D. Tracey
- David Wolpert
Organizations
- Air Force Office of Scientific Research
- Foundational Questions Institute
- National Science Foundation