DOM-based Content Extraction of HTML Documents

Abstract

Web pages often contain clutter (such as pop-up ads, unnecessary images and extraneous links) around the body of an article that distract a user from actual content. Extraction of "useful and relevant" content from web pages has many applications, including cell phone and PDA browsing, speech rendering for the visually impaired, and text summarization. Most approaches to removing clutter or making content more readable involve changing font size or removing HTML and data components such as images, which takes away from a webpage's inherent look and feel. Unlike "Content Reformatting", which aims to reproduce the entire webpage in a more convenient form, our solution directly addresses "Content Extraction". We have developed a framework that employs an easily extensible set of techniques that incorporate advantages of previous work on content extraction. Our key insight is to work with the Document Object Model tree, rather than with raw HTML markup. We have implemented our approach in a publicly available Web proxy to extract content from HTML web pages.

Open PDF

Document Details

Document Type
Technical Report
Publication Date
Jan 01, 2005
Accession Number
ADA437440

Entities

People

  • David Neistadt
  • Gail Kaiser
  • Peter Grimm
  • Suhit Gupta

Organizations

  • Columbia University

Tags

DTIC Thesaurus Topics

  • Automated Text Summarization
  • Computer Programming
  • Computers
  • Extraction
  • Graphical User Interface
  • Information Retrieval
  • Language
  • Mobile Phones
  • Natural Language Processing
  • Natural Languages
  • New York
  • Operating Systems
  • Removal
  • Standards
  • User Interface
  • Web Browsers
  • Websites

Fields of Study

  • Computer science

Readers

  • Agent-Based Social Robotics and Mobile-Assisted Learning in Virtual Environments.
  • Database Systems and Applications
  • Educational Psychology