Hallmarks of Human-Machine Collaboration: A Framework for Assessment in the DARPA Communicating with Computers Program

Abstract

There is a growing desire to create computer systems that can communicate effectively to collaborate with humans on complex, open-ended activities, often with a particular goal. Assessing this type of system presents significant challenges. This report describes a framework for evaluating collaborative computer systems engaged in open-ended complex scenarios where evaluators do not have the luxury of comparing performance to a single right answer. This framework was developed in the context of the DARPA Communicating with Computers (CwC) program, and has been used in the evaluation of communication in disparate human-machine creative collaborations, including story and music generation, interactive block building, and exploration of molecular mechanisms in cancer.

Open PDF

Document Details

Document Type
Technical Report
Publication Date
Jan 01, 2021
Accession Number
AD1123313

Entities

People

  • Bradley A. Goodman
  • Cheryl Clark
  • Christopher D. Garay
  • John S. Aberdeen
  • Lynette Hirschman
  • Matthew W. Peterson
  • Patricia L. Mcdermott
  • Robyn A. Kozierok
  • Tonia M. Korves

Organizations

  • MITRE Corporation

Tags

Communities of Interest

  • Autonomy

DTIC Thesaurus Topics

  • Computers
  • Contracts
  • Corporations
  • Demographic Cohorts
  • Human-Machine Interfaces
  • Human-Machine Systems
  • Language
  • Natural Languages
  • Teamwork
  • Test And Evaluation

Fields of Study

  • Computer science
  • Physics

Readers

  • Agent-Based Social Robotics and Mobile-Assisted Learning in Virtual Environments.
  • Instructional Design and Training Evaluation.