And Eat It Too: High Read Performance in Write-Optimized HPC I/O Middleware File Formats
Abstract
The two I/O middleware layer examined in this paper are the Adaptable IO System (ADIOS), a library-based approach developed at Oak Ridge National Laboratory to provide a high-level IO API that can be used in place of netCDF or HDF5 to do much more aggressive write-behind and efficient reordering of data locations within the file; and the Parallel Log-structured Filesystem (PLFS), a stackable FUSE filesystem approach developed at Los Alamos National Laboratory that decouples concurrent writes to improve the speed of checkpoints. We observe that not only can write-optimized I/O middleware be built to not penalize read speeds, but for important workloads, techniques that improve write performance can, perhaps counterintuitively, improve read speeds over reading from a contiguously organized file format. In the remainder of this paper, we investigate this further through case studies of PLFS and ADIOS on simulation checkpoint restarts.
Document Details
- Document Type
- Technical Report
- Publication Date
- Nov 15, 2009
- Accession Number
- ADA512866
Entities
People
- Garth Gibson
- Jay Lofstead
- John Bent
- Karsten Schwan
- Manish Parashar
- Matthew T Wolf
- Meghan Wingate
- Milo Polte
- Norbert Podhorszki
- Qing Liu
- Scott A. Klasky
Organizations
- Carnegie Mellon University