Closed-Loop Uncertainty: Evaluation of Uncertainty as a Probability of Machine Correctness
Abstract
Uncertainty values reported by machine learning models do not often accurately predict the probability of model correctness. Especially for interactive machine learning applications where curated labeled data may be scarce and data streams change modality, the conventional use of data-model statistics often fall short. This may be due in large part to the absence of feedback of correctness, which is feasible for a large number of human-in-the-loop applications. In this study, we explore the potential of incorporating such feedback into an interactive machine learning model for a threshold selection problem. The problem involves a user subjectively selecting the point at which they consider a signal to transition to a low state without providing specific rules to the user for what constitutes a low state. The classifier-driven machine model will attempt to mimic the analysts selection, with the signal becoming more complex in time. A methodology for evaluating uncertainty, which we define as the probability of machine correctness, is presented and used to compare a baseline model using naive Bayes with a novel reinforcement learning approach. The novel approach refines a black-box model for uncertainty by incorporating machine performance as feedback. Experiments are conducted over a large number of realizations in order to properly evaluate uncertainty using a stochastic process. Results show that our novel approach, called closed-loop uncertainty (CLU), outperforms the baseline in every case, yielding about 46 percent improvement over the baseline on average.
Document Details
- Document Type
- Technical Report
- Publication Date
- Nov 13, 2023
- Accession Number
- AD1215896
Entities
People
- Christopher Michael
- Zachary Bishof
Organizations
- United States Naval Research Laboratory