Shannon Inspired Approach to Limits of Learning
Abstract
1- We develop a new theoretical formulation of the limits of supervised learning when the objective of the learner is given by a parametric loss function and her goal is to assess the generalization capability of her inferred parametric function. 2- We present a numerical method for estimating the limits of learning in terms of generalization performance. We tailor the method to linear regression and logistic regression and calculate the learning limits in these scenarios. We compare the developed numerical framework with the existing widely used cross validation technique and demonstrate superior performance with significantly less computation. 3- We are currently studying the statistical properties of our developed generalization performance estimator, such as its variance, and its asymptotic distribution, in addition to extending the calculations to more complex problems, such as deep neural networks. 4- We are investigating neural circuits in the exacting setting that (i) knowledge acquisition can occur from single interactions, (ii) the results of these acquisitions are rapidly evaluable subcircuits, and (iii) recall in response to an external input can be in the form of a rapid evaluation of a composition of subcircuits that have been acquired at arbitrary different earlier times. 5- We are developing efficient algorithms for discovering multiple optima in Bayesian non-negative matrix factorization (NMF). We are currently focused on the question of characterizing the space of non-identifiability within a connected region using rapidly-exploring random trees.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Sep 11, 2018
- Source ID
- W911NF1610561
Entities
People
- Vahid Tarokh
Organizations
- Army Contracting Command
- Defense Advanced Research Projects Agency
- Harvard University