Metamodeling Techniques for Verification and Validation of Modeling and Simulation Data
Abstract
Modeling and simulation (M and S) outputs help the Director, Operational Test and Evaluation (DOT and E), assess the effectiveness, survivability, lethality, and suitability of systems. To use M and S outputs, DOT and E needs models and simulators to be sufficiently verified and validated. The purpose of this paper is to improve the state of verification and validation by recommending and demonstrating a set of statistical techniques metamodels, also called statistical emulators to the M and S community. The paper expands on DOT and Es existing guidance about metamodel usage by creating methodological recommendations the M and S community could apply to its activities. For a deterministic, discrete response variable, we recommend using a nearest neighbor or decision tree model. For a deterministic, continuous response variable, we recommend Gaussian process interpolation. For a stochastic response variable, we recommend a generalized additive model. We also present a set of techniques that testers can use to assess the adequacy of their metamodels. We conclude with a notional example (a paper plane simulation) that demonstrates the recommended techniques. Finally, we include supplemental software written in R that readers can use to reproduce the outputs from this paper.
Document Details
- Document Type
- Technical Report
- Publication Date
- Sep 01, 2022
- Accession Number
- AD1222943
Entities
People
- Curtis G. Miller
- John T. Haman
Organizations
- Institute for Defense Analyses