Design of Optimal Loss Functions for Statistical Estimation
Abstract
Statistical estimation aims at reconstructing an unknown object (an image, a signal, the position of an obstacle) from noisy or incomplete observations. Empirical risk minimization is (by far) the most popular approach to this problem. The unknown signal is estimated by minimizing a risk function, which is the sum of a loss term corresponding to each observation (plus, eventually, a regularizer). This project focuses on the optimal design of these loss terms, as to maximize reconstruction accuracy. Classical statistical theory (dating back to the first half of the 20th century) supports the use of log-likelihood loss. The fundamental justification for this choice is based on the low-dimensional asymptotics in which the number of unknown parameters d is fixed, while the number of observations n diverges. Modern applications to data analysis or signal processing require to estimate high-dimensional unknown signals. In these applications, the number of unknowns is typically of the same order as the number of observations. While classical low dimensional asymptotics does not apply to this regime, log-likelihood loss (and its variants) are still the method ofchoice. This project develops a new approach towards the design of loss functions, that aims at achieving minimax optimal reconstruction accuracy. The basic idea is that good loss functions should be tractable approximations of thefree energy (Legendre dual of the log-moment generating function). The log-likelihood loss can be viewed as an approximation of the free energy, which is accurate in the low-dimensional limit. The PI and collaborators will study anarray of free energy approximations that are better suited to the high-dimensional regime. This approach builds on ideas from statistical mechanics, as well as recent breakthroughs in probability theory. Newly constructed lossfunctions will be investigated from several points of view: (i) Establish statistical optimality in the high-dimensional regime; (ii) Develop efficient algorithms to optimize the new loss functions (which are often non-convex); (iii) Developversions of the same loss functions that are robust to model misspecification and outliers. This work will impact a wide range of signal processing and imaging tasks, including compressed sensing, phase retrieval, low rank signal estimation, as well as network estimation.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Jul 27, 2018
- Source ID
- N000141812729
Entities
People
- Andrea Montanari
Organizations
- Office of Naval Research
- Stanford University
- United States Navy