Huge Data Sets and the Frontiers of Computational Feasibility.
Abstract
Recently, Huber offered a taxonomy of data set sizes ranging from tiny (100 bytes) to huge (10 to the 10th power bytes). This taxonomy is particularly appealing because it quantifies the meaning of tiny, small, medium, large and huge. Indeed, some investigators consider 300 small and 10,000 large while others consider 10,000 small. In Huber's taxonomy, most statistical and visualization techniques are computationally feasible with tiny data sets. However with larger data sets computers run out of computational horsepower and graphics displays run out of resolution fairly quickly. In this paper, we discuss aspects of data set size and computational feasibility for general classes of algorithms in the context of CPU performance, memory size, hard disk capacity, screen resolution and massively parallel architectures. We discuss some strategies such as recursive formulations which mitigate the impact of size. We also discuss the potential for scalable parallelization which will mitigate the effects of computational complexity. (AN)
Document Details
- Document Type
- Technical Report
- Publication Date
- Nov 01, 1994
- Accession Number
- ADA291384
Entities
People
- Edward Wegman
Organizations
- George Mason University