Context-Dependent Conflation, Text Filtering and Clustering
Abstract
The presence of trivial words in text databases can impact record or concept (words or phrases) clustering adversely. Additionally, the determination of whether a word or phrase is trivial is context-dependent. The objective of the present paper is to demonstrate a context-dependent trivial word filter to improve clustering quality. Factor analysis was used as a context-dependent trivial word filter for subsequent term clustering. Medline records for Raynaud's Phenomenon were used as the database, and words were extracted from the record abstracts. A factor matrix of these words was generated, and the words that had low factor loadings across all factors were identified and eliminated. The remaining words, which had high factor loading values for at least one factor and therefore were influential in determining the theme of that factor, were input to the clustering algorithm. Both quantitative and qualitative analyses were used to show that factor matrix filtering leads to higher quality clusters and subsequent taxonomies. Additionally, a fractals database obtained from the Science Citation Index was used to demonstrate the value of factor matrices to determine the interchangeability of word variants, and show the context dependency requirements for conflation. (8 figures, approx. 90 refs.)
Document Details
- Document Type
- Technical Report
- Publication Date
- Sep 01, 2004
- Accession Number
- ADA426072
Entities
People
- Joel A. Block
- Ronald Neil Kostoff
Organizations
- Office of Naval Research