Portable Map-Reduce Utility for MIT SuperCloud Environment
Abstract
The MIT Map-Reduce utility has been developed and deployed on the MIT SuperCloud to support scientists and engineers at MIT Lincoln Laboratory. With the MIT Map-Reduce utility, users can deploy their applications quickly onto the MIT SuperCloud infrastructure. The MIT Map-Reduce utility can work with any applications without the need for any modifications. For improved performance, the MIT Map-Reduce utility provides an option to consolidate multiple input data files per compute task as a single stream of input with minimal changes to the target application. This enables users to reduce the computational overhead associated with the cost of multiple application starting up when dealing with more than one piece of input data per compute task. With a small change in a sample MATLAB image processing application, we have observed approximately 12x speed up by reducing the application startup overhead. Currently the MIT Map-Reduce utility can work with several schedulers such as SLURM, Grid Engine and LSF.
Document Details
- Document Type
- Technical Report
- Publication Date
- Sep 17, 2015
- Accession Number
- AD1034602
Entities
People
- Albert I. Reuther
- Andrew J. Prout
- Antonio Rosa
- Chansup Byun
- Charles Yee
- David Bestor
- Jeremy Kepner
- Julie Mullen
- Matthew Hubbell
- Peter W. Michaleas
- Vijay N. Gadepally
- William Arcand
- William Bergeron
Organizations
- MIT Lincoln Laboratory