Developing A Scaled Performance Evaluation Measurement System To Evaluate Marine Performance
Abstract
Training developers lack methods for determining the benefits of integrating live, virtual, and constructive training. This study defined and tested a scaled performance evaluation measurement system (SPEMS) to be used across tasks. We used the buddy rush task to test SPEMS and compare it to the current Go/No Go performance evaluation checklist (PECL). We developed SPEMS in three steps: we convened focus groups to establish five-level behaviorally anchored rating scales (BARS); confirmed SPEMS reliability using subject-matter expert (SME) virtual video analysis; and empirically tested SPEMS predictive capability in an operational environment. Suitable inter-rater reliability was found for BARS (87% agreement) and SPEMS (Cronbachs Alpha 0.93 to 0.98). Percent exposure was selected by SMEs as the objective measure of buddy rush performance. Fifty-two trainees (26 pairs) were evaluated using a PECL and SPEMS at three time points. The results showed that SPEMS has a moderate, negative, linear relationship with percent exposure at an R2 = 0.41/0.40. Conversely, PECL has a weak, slightly negative linear relationship with percent exposure at an R2 = 0.03/0.2. We reject the null hypotheses and conclude that SPEMS scores are significantly related to percent exposure and have more predictive strength than PECL scores. These findings demonstrate a verifiable, repeatable, and reliable potential solution to the problem of measuring military task performance across training solutions.
Document Details
- Document Type
- Technical Report
- Publication Date
- Jun 01, 2019
- Accession Number
- AD1080318
Entities
People
- Garrett A. Loeffelman
Organizations
- Naval Postgraduate School