Statistical Learning with large parameter spaces: Interpretable Nonparametrics, Conditional Computing and Beyond
Abstract
The proliferation of data from complex heterogeneous sources in varying shapes, forms and sizes, has led to a burning need for princ ipled tools in statistical inference to assimilate, understand and extract meaningful information from data. The current proposal ai ms to make new methodological contributions towards a better statistical and computational understanding of non-standard statistical learning problems that involve extremely large parameter spaces possibly endowed with a combinatorial structure. The first part of the project is on interpretable high-dimensional nonparametric regression problems with combinatorial shape constraints, and learnin g structured permutations arising in record linkage, supervised learning with mismatched data, etc. The second part explores the met hodological foundations of learning combinatorial objects such as trees via differentiable optimization methods to facilitate large- scale end-to-end continuous training. While trees are conventionally trained via greedy optimization methods, their relative merits and drawbacks compared to continuous optimization methods remain to be explored at a deeper level; and will be studied herein. This will pave the way towards studying the methodological foundations of conditional computing, a recent line of work that responds to severe computational challenges arising in massive neural network models with billions-trillions of parameters. A major computationa l bottleneck in such neural network models arises from dense computation where every sample propagates through all the nodes of the network. Conditional computing is a promising empirical approach that attempts to overcome this challenge by allowing every sample t o activate only a part of the network, similar to a decision tree. This project aims to initiate a formal study of conditional compu ting under a principled statistical and computational footing. In terms of technical approach, this project will draw upon foundatio nal principles in statistics and mathematical optimization. The current project will build upon recent algorithms created by the PI and collaborators in the context of sparse high-dimensional regression, and extend them in nontrivial ways to address problems in in terpretable nonparametrics and learning structured permutations. Related insights will play a critical role in understanding statist ical and computational tradeoffs in continuous and discrete optimization principles in conditional computing---with the goal of maki ng neural networks energy efficient. The Office of Naval Research is interested in creating new methods that involve processing larg e and small datasets of different types (image, text, documents, networks, etc) in times relevant for the application. A mission of ONR is to advance basic scientific research at the interface of mathematics, statistics, machine learning, mathematical optimization and related disciplines so that tools created can better inform complex decision-making processes, especially under uncertainty. Th e current proposal will study fundamental methods that play a crucial role in analyzing data from varied sources. This research is p oised to propose a fresh new perspective into many problems that are usually addressed with ad-hoc alternatives. This will address t he Navy and Department of Defense needs in statistical computation and modeling that play a crucial role in enabling automatic, robu st, accurate, and rapid decision-making.Approved for Public Release
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Aug 20, 2021
- Source ID
- N000142112841
Entities
People
- Rahul Mazumder
Organizations
- Massachusetts Institute of Technology
- Office of Naval Research
- United States Navy