Evaluating Message Understanding Systems: An Analysis of the Third Message Understanding Conference (MUC-3)

Abstract

This paper describes and analyzes the results of the Third Message Understanding Conference (MUC-3). It reviews the purpose, history, and methodology of the conference, summarizes the participating systems, discusses issues of measuring system effectiveness, describes the linguistic phenomena tests, and provides a critical look at the evaluation in terms of the lessons learned. One of the common problems with evaluations is that the statistical significance of the results is unknown. In the discussion of system performance, the statistical significance of the evaluation results is reported and the use of approximate randomization to calculate the statistical significance of the results of MUC-3 is described.

Open PDF

Document Details

Document Type
Technical Report
Publication Date
Sep 01, 1993
Accession Number
ADA594300

Entities

People

  • David D. Lewis
  • Lynette Hirschman
  • Nancy Chinchor

Organizations

  • Leidos

Tags

Communities of Interest

  • Energy and Power Technologies
  • Human Systems
  • Weapons Technologies

DTIC Thesaurus Topics

  • Computational Linguistics
  • Computational Science
  • Construction
  • Databases
  • Debugging
  • Information Retrieval
  • Information Science
  • Language
  • Lessons Learned
  • Linguistics
  • Machine Learning
  • Natural Language Processing
  • Natural Languages
  • Software Testing
  • Terrorists
  • Test Methods
  • Test Sets

Fields of Study

  • Computer science

Readers

  • Academic Conference Management
  • Computational Linguistics
  • Regression Analysis.