Sample Size Requirements and Considerations for Models to Assess Human-Machine System Performance
Abstract
Hierarchical Linear Models (HLMs), also known as multi-level models, are an extension of multiple regression analysis and can aid in the understanding of human and machine workloads of a system. These models allow for prediction and testing in systems with hierarchies of two or more levels. The complex interrelated variability of these multi-level models exists in operational settings, such as the Air Force Distributed Common Ground System Full Motion Video (AF DCGS FMV) community which is composed of individuals (Level-1), groups (Level-2), units (Level-3), and organizations (Level-4). Through the development of sample size requirements and considerations for multi-level models, this research determined necessary requirements and strategies to assess human-machine system performance based requests for statistical testing and evaluation. This research compares sample size recommendations to previous Level-2 HLM recommendations and extends recommendations to Level-3 and Level-4 HLMs based on varying effect sizes, level predictor variance, and error variance scenarios. Depending on the application, results demonstrate that sample size requirements may be smaller than what literature previously reported. Further, when sample size requirements cannot be met, this research develops and assesses re-sampling methods as a means to augment small samples for estimation. The operational community, DCGS FMV, has a small population of 1,000 and limited access to analysts. This research provides distributions based on Subject Matter Expert (SME) opinion for re-sampling methods to the DCGS FMV Level-3 and Level-4 HLMs. These findings provide a foundation on which sample size recommendations and extrapolation techniques for HLM are made. Such recommendations and techniques with SME input provide simulation based HLM assessments that can give initial recommendations with limited associated cost, analyst and researcher time, and resources.
Document Details
- Document Type
- Technical Report
- Publication Date
- Sep 02, 2019
- Accession Number
- AD1084395
Entities
People
- Jennifer S. Lopez
Organizations
- Air Force Institute of Technology