Evaluating Subjective Feedback for Internet of Things Dialogues

Abstract

This paper discusses the process of determining which subjective features are seen as ideal in a dialogue system, and linking these features to objectively quantifiable behaviors. A corpus of simulated system-user dialogues in the Internet of Things domain was manually annotated with a set of system communicative and action responses, and crowd-sourced ratings and qualitative feedback of these dialogues were collected. This corpus of subjective feedback was analyzed, revealing that raters described top ranked dialogues as Intelligent, Natural, Pleasant, and as having Personality. Additionally, certain communicative and action responses were statistically more likely to be present in dialogues described as having these features. There was also found to be a lack of agreement among raters as to whether a direct communication style, or a conversation alone was preferred, suggesting that future research and development should consider creating models for different communication styles.

Open PDF

Document Details

Document Type
Technical Report
Publication Date
Jan 01, 2018
Accession Number
AD1158500

Entities

People

  • Carla Gordon
  • David R Traum
  • Georgila Kallirroi
  • Hyungtak Choi
  • Jill Boberg

Organizations

  • University of Southern California

Tags

Communities of Interest

  • Autonomy

DTIC Thesaurus Topics

  • Artificial Intelligence
  • Cognitive Systems Engineering
  • Computational Linguistics
  • Computers
  • Dialogue Systems
  • Feedback
  • Information Systems
  • Internet
  • Internet Of Things
  • Judgment
  • Language
  • Linguistics
  • Mobile Devices
  • Natural Language Processing
  • Natural Languages
  • Personality
  • Statistical Analysis
  • Statistics

Readers

  • Agent-Based Social Robotics and Mobile-Assisted Learning in Virtual Environments.
  • Computational Modeling and Simulation
  • Organizational Psychology.

Technology Areas

  • 5G
  • 5G - Internet of Things