Large-Scale Exploratory Analysis, Cleaning, and Modeling for Event Detection in Real-World Power Systems Data

Abstract

In this paper, we present an approach to large-scale data analysis, Divide and Recombine (D&R), and describe a hardware and software implementation that supports this approach. We then illustrate the use of D&R on large-scale power systems sensor data to perform initial exploration discover multiple data integrity issues, build and validate algorithms to filter bad data, and construct statistical event detection algorithms. This paper also reports on experiences using a non-traditional Hadoop distributed computing setup on top of a HPC computing cluster.

Open PDF

Document Details

Document Type
Technical Report
Publication Date
Nov 01, 2013
Accession Number
ADA596393

Entities

People

  • Kerstin K Van Dam
  • Ryan Hafen
  • Tara D. Gibson
  • Terence Critchlow

Organizations

  • Pacific Northwest National Laboratory

Tags

Communities of Interest

  • Energy and Power Technologies
  • Materials and Manufacturing Processes

DTIC Thesaurus Topics

  • Algorithms
  • Computers
  • Data Analysis
  • Data Mining
  • Data Sets
  • Detection
  • Detectors
  • Electrical Grids
  • Engineering
  • Event Detection
  • High Performance Computing
  • Information Science
  • Load Monitoring
  • Measurement
  • Parallel Computing
  • Parallel Processing
  • White Noise

Fields of Study

  • Computer science
  • Engineering

Readers

  • Distributed Systems and Data Platform Development