Modern Probabilistic Models for Modern Deep Learning
Abstract
Modern Probabilistic Models for Modern Deep LearningMany of the modern successes of machine learning (ML) have focused on methods of deeplearning (DL). Deep learning provides a mechanism for "black box prediction": it analyzes adataset of inputs and outputs, and produces a function to predict an output from an input. The DLparadigm is successful because it is both expressive and scalable. It can capture complexrelationships between input and output, and the ML research community has developed efficientalgorithms and software to develop and deploy DL methods with massive datasets.But while black-box prediction through DL has been be empirically proven, it is also limited. First,DL does not involve clearly articulated assumptions. Though it is now easy to apply a DL methodto a dataset, it is not clear what statistical assumptions it is making, and where the data satisfiesthem. Second, basic DL methods provide point predictions, but do not help estimate uncertaintyabout them. Among a set of predictions, an analyst cannot know which ones are confidently madeand which ones are close to guesses. Finally, DL does not provide interpretable predictions, inthat it does not reveal why or how it has made its inferences.To overcome these limitations, we will turn to the rigorous methodology of probabilistic ML andapplied Bayesian statistics. In the paradigm of probabilistic ML, we frame each ML problem witha formal probabilistic model, one that contains hidden and observed random variables. We thenconsider inference and prediction as calculations of conditional distributions within the model. Aswe will see, probabilistic ML does not suffer from the same limitations as deep learning. Itprovides easy ways to encode assumptions and theories into machine learning algorithms; itprovides a natural mechanism for measuring uncertainty about the resulting predictions; and itprovides clear ways to interpret and explain its inferences and results.Thus our approach is to understand and re-interpret deep learning through the lens of probabilisticmachine learning, and to develop new methods that are a fusion of the best ideas of both fields.The results of this research are algorithms and applications that enjoy the expressivity andpredictive power of deep learning with the interpretability, robustness, and clarity of probabilisticML. In addition to providing foundational advances to the science of ML, this research will impactDoD capabilities, particularly those around fusing disparate sources of information, interpretablerepresenting and reasoning about high-dimensional signals, and forming complex predictions fromdata.The proposed research is organized into three related research thrusts. Each one centers around afusion of ideas from deep learning and probabilistic ML. (1) We will study canonical linkfunctions in a Bayesian generalization of sigmoid belief networks, generalizing classical ML ideaswith statistical ideas like exponential families and generalized linear models. (2) We will developBayesian nonparametric deep learning to help define ``infinite-depth models that naturallycapture fidelity to the training data with protection against overfitting. (3) We will developempirical Bayes (EB) algorithms for representation learning, one that draws from Bayesianstatistics, frequentist statistics, and information theory.This abstract is approved for public release.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Jan 06, 2021
- Source ID
- N000142112120
Entities
People
- David M. Blei
Organizations
- Office of Naval Research
- Trustees of Columbia University in the City of New York
- United States Navy