Sparse Forward-Backward for Fast Training of Conditional Random Fields
Abstract
Complex tasks in speech and language processing often include random variables with large state spaces, both in speech tasks that involve predicting words and phonemes, and in joint processing of pipelined systems in which the state space can be the labeling of an entire sequence. In large state spaces, however, discriminative training can be expensive, because it often requires many calls to forward-backward. Beam search is a standard heuristic for controlling complexity during Viterbi decoding, but during forward-backward, standard beam heuristics can be dangerous, as they can make training unstable. The authors introduce sparse forward-backward, a variational perspective on beam methods that uses an approximating mixture of Kronecker delta functions. This motivates a novel minimum-divergence beam criterion based on minimizing Kullback-Leibler (KL) divergence between the respective marginal distributions. This beam selection approach is not only more efficient for Viterbi decoding, but also more stable within sparse forward-backward training. For a standard text-to-speech problem, they reduce CRF training time fourfold -- from over a day to 6 hours -- with no loss in accuracy.
Document Details
- Document Type
- Technical Report
- Publication Date
- Jan 01, 2006
- Accession Number
- ADA443633
Entities
People
- Andrew McCallum
- Charles Sutton
- Chris Pal
Organizations
- University of Massachusetts Amherst