Users' Manual and Validation of the Automated Grading System (AGS): Improving the Quality of Intelligence Summaries Using Feedback from an Unsupervised Model of Semantics
Abstract
The Automated Grading System (AGS) was developed jointly by Defence Research and Development Canada (DRDC) Toronto and the Canadian Forces School of Military Intelligence (CFSMI) to provide students at the school a tool to help in the composition of accurate and effective Intelligence Summaries (INTSUMs). The AGS is a web-browser based system that provides feedback to students about how well their summary matches that of a gold standard summary written by an instructor. The AGS allows students to iteratively correct and re-submit their summaries as they attempt to maximize the match between their summary and the gold standard. In this report, we provide both the instructor and student user's manual for the AGS. Importantly, we also provide the results of a small validation study wherein we asked participants to summarize news stories about sea piracy near Somalia. Participants used feedback from the AGS to improve their summaries until they were satisfied that they had done the best job they could do. The grades given to the first and final summaries by the AGS were then compared to the grades awarded by the lead instructor at CFSMI. The tool and the instructor's assessments of the summaries were in close agreement. The results confirm that the AGS can be used as an effective teaching tool to help students improve their summary-writing skills.
Document Details
- Document Type
- Technical Report
- Publication Date
- Dec 01, 2012
- Accession Number
- ADA573145
Entities
People
- Benjamin Stone
- Peter Kwantes
- Ron Wulf
Organizations
- Defence Research and Development Canada