A Fast Variational Approach for Learning Markov Random Field Language Models

Abstract

Language modelling is a fundamental building block of natural language processing. However in practice the size of the vocabulary limits the distributions applicable for this task: specifically one has to either resort to local optimization methods, such as those used in neural language models, or work with heavily constrained distributions. In this work, we take a step towards overcoming these difficulties. We present a method for global-likelihood optimization of a Markov random field language model exploiting long-range contexts in time independent of the corpus size. We take a variational approach to optimizing the likelihood and exploit underlying symmetries to greatly simplify learning. We demonstrate the efficiency of this method both for language modelling and for part-of-speech tagging.

Open PDF

Document Details

Document Type
Technical Report
Publication Date
Jan 01, 2015
Accession Number
ADA621682

Entities

People

  • Alexander M. Rush
  • David A Sontag
  • Yacine Jernite

Organizations

  • New York University

Tags

Communities of Interest

  • Autonomy

DTIC Thesaurus Topics

  • Air Force Research Laboratories
  • Algorithms
  • Artificial Intelligence
  • Artificial Intelligence Software
  • Computational Linguistics
  • Computational Science
  • Computer Languages
  • Data Sets
  • Information Processing
  • Language
  • Machine Learning
  • Natural Language Processing
  • Natural Languages
  • Optimization
  • Probability
  • Statistical Analysis
  • Vocabulary

Fields of Study

  • Computer science

Readers

  • Computational Linguistics
  • Neural Network Machine Learning.
  • Systems Analysis and Design

Technology Areas

  • AI & ML
  • AI & ML - Machine Learning Algorithms
  • AI & ML - Machine Translation