Programming Abstractions for Managing Workflows on Tiered Storage Systems
Abstract
Scientific workflows in High Performance Computing ( HPC ) environments are processing large amounts of data. The storage hierarchy on HPC systems is getting deeper, driven by new technologies (NVRAMs, SSDs, etc.) There is a need for new programming abstractions that allow users to seamlessly manage data at the workflow level on multi-tiered storage systems, and provide optimal workflow performance and use of storage resources. In previous work, we introduced a software architecture Managing Data on Tiered Storage for Scientific Workflows (MaDaTS ) that used a Virtual Data Space ( VDS ) abstraction to hide the complexities of the underlying storage system while allowing users to control data management strategies. In this article, we detail the data-centric programming abstractions that allow users to manage a workflow around its data on the storage layer. The programming abstractions simplify data management for scientific workflows on multi-tiered storage systems, without affecting workflow performance or storage capacity. We measure the overheads and effectiveness introduced by the programming abstractions of MaDaTS. Our results show that these abstractions can optimally use the storage capacity in lesser capacity storage tiers, and simplify data management without adding any performance overheads.
Document Details
- Document Type
- Pub Defense Publication
- Publication Date
- Oct 25, 2021
- Source ID
- 10.1145/3457119
Entities
People
- Devarshi Ghoshal
- Lavanya Ramakrishnan
Organizations
- Lawrence Berkeley National Laboratory
- National Energy Research Scientific Computing Center
- Office of Advanced Scientific Computing Research
- Office of Science
- United States Department of Energy