Joint Experiment on Scalable Parallel Processors (JESPP) Parallel Data Management
Abstract
The need to present quantifiable results from simulations to support transformational finds is driving the creation of very large and geographically dispersed data collections. The US Joint Forces Command is conducting a series of Urban Resolve experiments to investigate concepts for applying future technologies to join urban warfare. The recently concluded experiments utilized and integrated multiple Scalable Parallel Processors (SPP) sites distributed across the United States. This computational power is required to model futuristic sensor technology and the complexity of urban environments. The Urban Resolve simulation generated more than two terabytes of raw data at a rate of >10 gigabytes per hour. The size and distributed nature of this type of data collection pose significant challenges in developing the corresponding data-intensive applications that manage and analyze them. We present here a next generation data management and analysis tool, called Simulation Data Grid (SDG). The design principles driving the design of SDG are: 1) minimize network communication overhead by storing data near the point of generation and only selectively propagating the data as needed, and 2) maximize the use of SPP computational resources and storage by distributing analyses across SPP sites to reduce, filter and aggregate the data.
Document Details
- Document Type
- Technical Report
- Publication Date
- May 01, 2006
- Accession Number
- ADA451492
Entities
People
- Dan M. Davis
- Robert F. Lucas
Organizations
- University of Southern California