ILR-Based MT Comprehension Test with Multi-Level Questions

Abstract

We present results from a new Interagency Language Roundtable (ILR) based comprehension test. This new test design presents questions at multiple ILR difficulty levels within each document. We incorporated Arabic machine translation (MT) output from three independent research sites, arbitrarily merging these materials into one MT condition. We contrast the MT condition, for both text and audio data types, with high quality human reference Gold Standard (GS) translations. Overall, subjects achieved 95% comprehension for GS and 74% for MT, across all genres and difficulty levels. Interestingly, comprehension rates do not correlate highly with translation error rates, suggesting that we are measuring an additional dimension of MT quality.

Open PDF

Document Details

Document Type
Technical Report
Publication Date
Apr 01, 2007
Accession Number
ADA507068

Entities

People

  • Arvind Jairam
  • Douglas Jones
  • Edward Gibson
  • Hussny Ibrahim
  • Martha Herzog
  • Michael Emonts
  • Wade Shen

Organizations

  • Massachusetts Institute of Technology

Tags

DTIC Thesaurus Topics

  • Abstracts
  • Accuracy
  • Arabic Language
  • Comprehension
  • Computational Linguistics
  • Construction
  • Errors
  • Foreign Languages
  • Language
  • Linguistics
  • Machine Translation
  • Materials
  • Natural Language Processing
  • Standards
  • Test And Evaluation
  • Test Methods
  • Translations

Fields of Study

  • Computer science

Readers

  • Aerospace Test and Evaluation
  • Computational Linguistics
  • Instructional Design and Training Evaluation.

Technology Areas

  • AI & ML
  • AI & ML - Bayesian Inference
  • AI & ML - Information Retrieval
  • AI & ML - Machine Translation