Plans for a Task-Oriented Evaluation of Natural Language Understanding Systems

Abstract

A plan is presented for evaluating natural language processing (NLP) systems that have focused on the issues of text understanding as exemplified in short texts from military messages. The plan includes definition of bodies of text to use as development and test data, namely the narrative lines from one type of naval message, and definition of a simulated database update task that requires NLP systems to fill a template with information found in the texts. Documentation related to the naval messages and examples of filled templates have been prepared to assist NLP system developers. It is anticipated that developers of a number of different NLP systems will participate in the evaluation and will meet afterwards to present and interpret the results and to critique the test design.

Open PDF

Document Details

Document Type
Technical Report
Publication Date
Jan 01, 1989
Accession Number
ADA458875

Entities

People

  • Beth M. Sundheim

Tags

Communities of Interest

  • Air Platforms
  • Ground and Sea Platforms
  • Materials and Manufacturing Processes
  • Space
  • Weapons Technologies

DTIC Thesaurus Topics

  • Abstracts
  • Construction
  • Database Management Systems
  • Databases
  • Formal Languages
  • Instructions
  • Language
  • Measurement
  • Natural Language Processing
  • Natural Language Understanding
  • Natural Languages
  • Navy
  • Targets
  • Template Patterns
  • Test And Evaluation
  • Test Sets

Fields of Study

  • Computer science

Readers

  • Computational Linguistics
  • Software Engineering
  • Software Engineering.

Technology Areas

  • AI & ML
  • AI & ML - DoD AI Strategy
  • AI & ML - Machine Translation