A Tool for Creating and Parallelizing Bioinformatics Pipelines
Abstract
Bioinformatics pipelines enable hfr scientists to effectively analyze biological data through automated multi-step processes constructed by individual programs and databases. The huge amount of data and time consuming computations require effectively parallelized pipehnes to provide results within a reasonable time. To reduce researchers' programming burden for pipeline creation and parallelization, we developed the Bioinformatics Pipeline Generation and Parallelization Toolkit (B io Gent). A user needs only to create a pipehne definition file that describes the data processing sequence and input/output files. A program termed schedpipe in the BioGent toolkit takes the definition file and executes the designed procedure. Schedpipe automatically parallelizes the pipeline execution by performing independent data processing steps on muliple CPUs, and by decomposing big datasets into small chunks and processing them in parallel. Schedpipe controls program execution on multiple CPUs through a simple application programming interface (API) of the Parallel Job Manager (PJM) library. As a part of the BioGent toolkit, PJM was developed to effectively launch and monitor programs on multiple CPUs using a Message Passing Interface (MPI) protocol. The PJMAPI can also be used to parallelize other serial programs. A demonstration using PJM for parallelization shows 10% to 50% savings in time compared to an indigenous parallelization through a batch queuing system.
Document Details
- Document Type
- Technical Report
- Publication Date
- Jun 01, 2007
- Accession Number
- ADP023782
Entities
People
- Chenggang Yu
- Paul A. Wilson
Organizations
- United States Army Medical Research and Development Command