Metric Selection for Evaluation of Human Supervisory Control Systems
Abstract
Previous research has identified broad metric classes for human-automation performance in order to facilitate metric selection, as well as understanding and comparing research results. However, there is still a lack of a systematic method for selecting the most efficient set of metrics when designing experiments evaluating human-system performance. This report identifies and presents a list of evaluation criteria that can help determine the quality of a metric in terms of experimental constraints, comprehensive understanding, construct validity, statistical efficiency, and measurement technique efficiency. Based on these evaluation criteria, a comprehensive list of potential metric costs and benefits is generated. The evaluation criteria along with the list of metric costs and benefits, and the existing generic metric classes are then used to develop cost-benefit functions. Depending on research objectives and limitations, the entries in the cost and benefit functions can have different weights of importance. In order to help researchers assign subjective weights for these cost function criteria, two different multi-criteria decision making methods were investigated through an experiment with subject matter experts. These two methods are the analytic hierarchy process (AHP), and the ranking input matrix (RIM) method. Although RIM was preferred more than AHP, the results of the experiment did not reveal substantial benefits to either of the methods with respect to metric selection. The majority of participants' metric selections before using the methods were the same as the suggestions provided by AHP and/or RIM. However, RIM was more positively viewed than the AHP method. In addition, the majority of the participants rated the evaluation criteria used in both tools as very useful.
Document Details
- Document Type
- Technical Report
- Publication Date
- Dec 01, 2009
- Accession Number
- ADA530918
Entities
People
- Birsen Donmez
- M. L. Cummings
Organizations
- Massachusetts Institute of Technology