University of Massachusetts: MUC-3 Test Results and Analysis
Abstract
We believe that the score reports we obtained for TST2 provide an accurate assessment of our system's capabilities insofar as they are consistent with the results of our own internal tests conducted near the end of phase 2. The required TSTh score reports indicate that our system achieved the highest combined scores for recall (51%) and precision (62%) as well as the highest recall score of all the MUC-3 systems under the official MATCHED/MISSING scoring profile. We ran one optional test in addition to the required test for TST2. The optional run differs from the required run in only one respect, an alteration to our consolidation module. The consolidation module contains all procedures that translate parser output into target template instantiations. The complete consolidation module includes a case-based reasoning (CBR) component that makes predictions about the target output based on a portion of the development corpus. For our optional run, we executed a modified version of consolidation that does not include this CBR component. We predicted that the absence of the CBR component would pull recall down but push precision up (looking at MATCHED/MISSING only). This trade off prediction was confirmed by the required and optional TST2 score reports. (Please consult Appendix F for our required and optional test score summaries).
Document Details
- Document Type
- Technical Report
- Publication Date
- Jan 01, 1991
- Accession Number
- ADA458572
Entities
People
- Claire Cardie
- David Fisher
- Ellen Riloff
- Robert Williams
- Wendy Lehnert
Organizations
- University of Massachusetts Amherst