A Framework for Assessing the Confidence in Human-Guided Modeling of Complex Engineering Systems
Abstract
Predictive models are powerful tools adopted for a broad range of Naval applications and include commonly adopted methods such as finite element models, as well as data-driven, machine learning models. While a tremendous amount of work has been done to build predictive models, the overwhelming measure of success is model accuracy, that is, did the model get the answer correct? Yet, quantification and communication of how incorrect a model might be or how uncertain a model#s output (sometimes known as predictive uncertainty) is not routinely considered.As models, by definition, are merely representations of physical systems and assets, the key guiding principle of the proposed work is to acknowledge that even models of the highest fidelity are still wrong. Even high-fidelity modelsare wrong because they require humans to make numerous design assumptions in the form of boundary conditions, input data, and in many software applications, choosing from a limited selection of discrete options. Some assumptions are quite close to those encountered in the real-world system or structure, whereas others may be quite far from reality but are made for model tractability reasons. Further, the human user typically has some knowledge or beliefs about which assumptions are least like reality versus the assumptions in which they have the most confidence. This is often highly valuable additional knowledge that is nonetheless discarded once the model is put into use despite its influence on ensuing predictions. This project creates a general framework that incorporates feedback from human users in a resource-efficient way to improve the uncertainty quantification estimates of model-based predictions forengineering design problems. A portfolio of expert elicitation protocols is created and then integrated into Bayesian belief networks (BBNs) that account for the reasoning that human experts rely upon during the conception, design, and construction of computational models. The cost-effectiveness of the elicitation protocols is determined though an integer programming-based optimization approach. This optimization model takes as input different elicitation protocols that require varying levels of human involvement but also different levels of uncertainty expression accuracy, and determines which type of elicitation protocol to use for which assumption, given a limited budget on human elicitation effort. The information collected from human experts is then used to estimate individual assumptions confidence measures (e.g., probability distribution functions, histograms of discretized measures of expert judgment, such as low-, medium-, high-confidence) that are subsequently integrated into BNNs. The BBNs provide a systematic measure thatthe model is correct given expert judgments about the many model inputs, and can be viewed as a measure of an expert s ultimate confidence in the model#s representative ability.This program proposes to explore the viability of the framework to quantify the uncertainty introduced by human decision-making through a case study of generating a finite element model of a ship structure. The ensuing computational experiments will help to quantify the value of different elicitation protocols, the value of the optimization approach, and the value of different BBN architectures for organizing that knowledge into confidence measures.APPROVED FOR PUBLIC RELEASE
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Nov 08, 2024
- Source ID
- N000142412418
Entities
People
- Patrick T Brewick
Organizations
- Office of Naval Research
- United States Navy
- University of Notre Dame