The RADAR Test Methodology: Evaluating a Multi-Task Machine Learning System with Humans in the Loop

Abstract

The RADAR (Reflective Agents with Distributed Adaptive Reasoning) project involves a collection of machine learning research thrusts that are integrated into a cognitive personal assistant. Progress is examined with a test developed to measure the impact of learning when used by a human user. Three conditions (conventional tools, Radar without learning, and Radar with learning) are evaluated in a large-scale, between-subjects study. This paper describes the RADAR Test with a focus on test design, test harness development, experiment execution, and analysis. Results for the 1.1 version of Radar illustrate the measurement and diagnostic capability of the test. General lessons on such efforts are also discussed.

Open PDF

Document Details

Document Type
Technical Report
Publication Date
Oct 01, 2006
Accession Number
ADA457300

Entities

People

  • Aaron Steinfeld
  • Dan Siewiorek
  • Django Wexler
  • Julie Fitzgerald
  • Kyle Cunningham
  • Matt Lahut
  • Othar Hansson
  • Pablo-alejandro Quinones
  • Paul Cohen
  • Rachael Bennett

Organizations

  • Carnegie Mellon University

Tags

Communities of Interest

  • Autonomy

DTIC Thesaurus Topics

  • Artificial Intelligence
  • Cognitive Systems Engineering
  • Computational Science
  • Computer Science
  • Electronic Mail
  • Human-Computer Interaction
  • Machine Learning
  • Multiagent Systems
  • Pattern Recognition
  • Psychological Phenomena And Processes
  • Psychology
  • Radar
  • Test And Evaluation
  • Test Methods
  • User Interface
  • Websites
  • Word Processors

Fields of Study

  • Computer science

Readers

  • Artificial Intelligence
  • Radar Systems Engineering.
  • Software Engineering

Technology Areas

  • AI & ML