New Directions in Conformal Inference: Modular Uncertainty Quantification for Complex Prediction Systems
Abstract
Approved for Public ReleaseLearning algorithms are increasingly embedded in real-world systems, but there has been little research o n tolerance regions or confidence regions for such algorithms. This makes it difficult to use these systems in high-stakes decision- making settings. Indeed, their use is currently principally restricted to making predictions. Decision-making requires uncertainty q uantification (UQ)knowing when a prediction engine will fail, and how badly.Contemporary automated systems demand diverse forms of data and ever-evolving learning strategies. In attempting to provide UQ in such settings, engineers must currently proceed with ad-h oc algorithms, with unclear performance and statistical foundations, or simply report point predictions together with an estimate of the overall model accuracy.As an illustrative example, when using a neural network to predict metabolic activity from medical image s, the current standard is to output a point prediction as well as the mean-squared error on a validation set, and go no farther to flag challenging instances with high predictive uncertainty. When deployed, the doctor does not know if the prediction is reliable f or any one input imageknowing the overall mean squared error may be of little comfort for making a decision about a high-stakes int ervention. This setting demands uncertainty measures tailored to each particular patient to provide credible information to aid the doctors treatment decision. Retreating to easy-to-analyze models is not the solution; rather, we need tools for UQ compatible with the entire spectrum of state-of-the-art models. Importantly, UQ should not interfere with the engineering choices that drive perform ance gains, and it must be able to handle complex inputs and outputs, such as video, 3D volumes, natural text, and so on. A line of work called Distribution-Free Uncertainty Quantification, exemplified by conformal inference, is an important first step in this dir ection. Conformal inference endows any predictive model with statistical guarantees, yielding confidence sets for predictions that c ertifiably contain the true label with user-specified probability, such as 90%. It provides finite-sample guarantees for any (unknow n) data-generating distribution and any predictive model, giving engineers trustworthy instance-wise uncertainty estimates for the f irst time. However, conformal inference can only handle one simple statistical error rate that does not apply to manyif not mostma chine learning settings. In this research proposal we present a new technique, inspired by conformal prediction but permitting arbit rary notions of statistical error, that provides UQ for a wide range of machine learning tasks, under no distributional assumptions, and with modest computational burden. Most notably, our method applies to non-monotone statistical quantities, such as the false-di scovery rate, and it reduces the UQ problem to a multiple hypothesis problem. In particular we study a calibration procedure that se lects from among many possible set-valued predictors and which adjusts for multiplicity. Establishing theoretical guarantees for thi s procedure will require new research on concentration inequalities for set-valued predictors.If successful this line of research pr omises to provide real-time assessments of accuracy and tolerance in time-critical statistical estimation for high-noise regimesa c lass of problems that are widespread in strategic and tactical decision-making.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Aug 20, 2021
- Source ID
- N000142112840
Entities
People
- Michael I. Jordan
Organizations
- Office of Naval Research
- United States Navy
- University of California Regents