BBN PLUM: MUC-3 Test Results and Analysis
Abstract
Perhaps the most important facts about our participation in MUC-3 reflect our starting point and goals. In March, 1990, we initiated a pilot study on the feasibility and impact of applying statistical algorithms in natural language processing. The experiments were concluded in March, 1991 and lead us to believe that statistical approaches can effectively improve knowledge-based approaches [Weishedel, et al., 1991a, Weischedel, Meteer, and Schwartz, 1991]. Due to nature of that effort, we had focused on many well-defined algorithm experiments. We did not have a complete message processing system; nor was the pilot study designed to create an application system. For the Phase I evaluation, we supplied a module to New York University. At the time of the Phase I Workshop (12-14 February 1991) we decided to participate in MUC with our own entry. The Phase I Workshop provided invaluable insight into what other sites were finding successful in this particular application. On 25 February, we started an intense effort not just to be evaluated on the FBIS articles, but also to create essential components (e.g., discourse component and template generator) and to integrate all components into a complete message processing system. Although the timing of the Phase II test (6-12 May) was hardly ideal for evaluating our site's capabilities, it was ideally timed to serve as a benchmark prior to starting a four year plan for research and development in message understanding. Because of this, we were determined to try alternatives that we believed would be different than those employed by other groups, wherever time permitted. These are covered in the next section.Our results were quite positive, given these circumstances. Our max-tradeoff version achieved 45% recall and 52% precision with 22% overgenerating (See Figure 2). PLUM can be run in several modes, trading off recall versus precision and overgeneration.
Document Details
- Document Type
- Technical Report
- Publication Date
- Jan 01, 1991
- Accession Number
- ADA460729
Entities
People
- Damaris Ayuso
- Jeff Palmucci
- Ralph Weischedel
- Robert Ingria
- Sean Boisen
Organizations
- BBN Technologies