Trusted Machine Learning: Statistical Tools for Making the Black Box Effective
Abstract
The ongoing data science revolution has been driven by impressive technological advances in the capture, storage, and processing of data, across a wide range of domains. Of particular interest is the recent progress in machine learning (ML) which provides us with many potentially effective tools to learn from datasets of ever increasing sizes and make useful predictions. These modelsrely less and less on principles from the physical world and increasingly on the amalgamation of large heterogeneous data sets. How do we know that these models can be trusted in critical and high-sensitivity systems? Our project will introduce novel ideas to ensure that the learned models satisfy some crucial properties; reliability, robustness, reproducibility, and fairness in the sense that the models need to apply to individuals in an equitable manner. To achieve these important objectives, the originality of our approach will be this: instead of ~opening up the black box~ and trying to understand its underpinnings, our project calls for the development of broad methodologies that can be wrappedaround any black box as to produce results that can be trusted. We will leverage recent progress on model-free predictive inference and on selective inference~most notably the knockoffs framework for replicable selections~to make progress in three directions. First, we will develop methods which rigorously assess the accuracy of any predictive engine, which is crucial in situationswhere the consequences of a decision seriously impact peoples~ lives. The target is the following rigorous objective: construct a predictive set as short as possible, which is guaranteed to contain the true (unknown) label with user-specified probability. Second, we will develop highly operational statistical tools as to provide decision makers with the most accurate facts and evidence,even though we may have unbalanced data sets; that is, even when we are presented with a biased view of the data because certain groups or cases are overly represented. Here, one objective is to provide decision makers with an intuitive measure of the limits of predictive performance of any algorithm. Another objective is to guarantee a use of data-driven recommendation systems supporting a fair and equitable treatment. Third, we shall introduce techniques to rigorously draw causal inferences~inferences provably immune to all confounding variables~from some observational data, and most notably from data found in genome-wide association studies. In particular, we shalldevelop methods that provably localize the causal genetic variants into explicit windows along the genome, drawing on recent technical advances in conditional hypothesis testing. In contrast, we will prove that classical methods produce spurious findings because of linkage disequilibrium. Machine learning algorithms/tools are part of large and complex systems used by the Navy.These systems routinely evaluate concrete situations and inform important decisions and actions. It is critical to have confidence in our decisions and actions, and to have accurate measures of potential losses. How do we make sure that the features extracted by various machine learning systems (e.g. deep neural networks) are reliable? How do we make sure that they can operate asa mission-critical component in a broader pipeline going from data to action? The tools from this proposal address such issues.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Feb 17, 2020
- Source ID
- N000142012157
Entities
People
- Emmanuel Candès
Organizations
- Office of Naval Research
- Stanford University
- United States Navy