An Innovative Approach for Assessing Knowledge in Air-to-Air Distributed Mission Training
Abstract
This paper highlights the development and lessons learned from a vignette-based approach to knowledge assessment. Our initial development which is based on Situational Judgment Inventory (SJI) and Job Knowledge Inventory (JKI) research, used an open-ended paper-based assessment instrument, referred to as Situation Assessment and Action Selection (SAAS), to examine pilots assessment of air-to-air situations as well as their opinions on appropriate courses of action. Scoring of pilot responses was challenging. One limiting factor in using open-ended responses is the time and effort required to score them. We are exploring the use of automated scoring of the responses, beginning with Latent-Semantic Analysis (LSA). Successful LSA scoring would greatly enhance the utility of the method and support the next phase of development. The next phase of development is intended to be a more automated version of the instrument, referred to as the Air Superiority Knowledge Assessment System (ASKAS). Results from our evaluation of SAAS are presented and discussed. Lessons learned and a rationale for developing a multimedia-based assessment system is discussed. Finally, key features of ASKAS are described with respect to their potential for helping researchers and practitioners assess the impact of DMT on pilots knowledge and understanding.
Document Details
- Document Type
- Technical Report
- Publication Date
- Dec 01, 2002
- Accession Number
- ADA532648
Entities
People
- Anna Castillo
- Brenda Wenzel
- Jack Wooster
- Michael Park
- Robert Robbins
- Roger Schvaneveldt
- Sarah Kotte
- Winston Bennett Jr.
Organizations
- Air Force Research Laboratory