Measuring Translation Quality by Testing English Speakers with a New Defense Language Proficiency Test for Arabic
Abstract
We present results from an experiment in which educated English-native speakers answered questions from a machine translated version of a standardized Arabic language test. We compare the machine translation (MT) results with professional reference translations as a baseline for the purpose of determining the level of Arabic reading comprehension that current machine translation technology enables an English speaker to achieve. Furthermore, we explore the relationship between the current, broadly accepted automatic measures of performance for machine translation and the Defense Language Proficiency Test, a broadly accepted measure of effectiveness for evaluating foreign language proficiency. In doing so, we intend to help translate MT system performance into terms that are meaningful for satisfying Government foreign language processing requirements. The results of this experiment suggest that machine translation may enable Interagency Language Roundtable Level 2 performance, but is not yet adequate to achieve ILR Level 3. Our results are based on 69 human subjects reading 68 documents and answering 173 questions, giving a total of 4,692 timed document trials and 7,950 question trials. We propose Level 3 as a reasonable near-term target for machine translation research and development.
Document Details
- Document Type
- Technical Report
- Publication Date
- May 01, 2005
- Accession Number
- ADA510883
Entities
People
- Clifford Weinstein
- Douglas Jones
- Martha Herzog
- Neil Granoien
- Wade Shen
Organizations
- Massachusetts Institute of Technology