Modeling and Interpreting Multimodal Inputs: A Semantic Integration Approach

Abstract

Modern user interfaces can take advantage of multiple input modalities such as speech, gestures, handwriting... to increase robustness and flexibility. The construction of such multimodal interfaces would be greatly facilitated by a unified framework that provides methods to characterize and interpret multimodal inputs. In this paper we describe a semantic model and a multimodal grammar structure for a broad class of multimodal applications. We also present a set of grammar-based Java tools that facilitate the construction of multimodal input processing modules, including a connectionist network for multimodal semantic integration.

Open PDF

Document Details

Document Type
Technical Report
Publication Date
Dec 01, 1997
Accession Number
ADA336561

Entities

People

  • Alex Waibel
  • Minh T. Vo

Organizations

  • Carnegie Mellon University

Tags

Communities of Interest

  • C4I

DTIC Thesaurus Topics

  • Algorithms
  • Automated Speech Recognition
  • Computer Programming
  • Computer Science
  • Computers
  • Computing System Architectures
  • Grammars
  • Graphical User Interface
  • Language
  • Models
  • Network Architecture
  • Probabilistic Models
  • Probability
  • Semantic Models
  • Statistical Samples
  • User Interface
  • Web Browsers

Readers

  • Agent-Based Social Robotics and Mobile-Assisted Learning in Virtual Environments.
  • Computational Linguistics
  • Speech Processing/Speech Recognition.