Stochastic Estimations of the Total Number of Classes for a Clustering having Extremely Large Samples to be Included in the Clustering Engine
Abstract
Numerous reports have elucidated the classification of a large amount of data using various clustering techniques. However, an increase in data size hinders the applicability of these methods. Here, it is investigated how to deal with the exploding number of possibilities to be sorted into irreducible classes by using a clustering technique when its input capacity cannot accommodate the total number of possibilities. This can be exemplified by atomic substitutions in the supercell modeling of alloys. The number of possibilities is sometimes equal to trillions, which is extremely large to be accommodated in a cluster. Thus, it is not practically feasible to identify directly how many irreducible classes exist even though several techniques are available to perform the clustering. In this regard, a stochastic framework is developed to avoid the shortage limitations, providing a method to estimate the total number of irreducible classes (the order of classes), as a statistical estimate. The main conclusion is that the statistical variation of the number of classes, at each sampling trial, can serve as a promising measure to estimate the total number of irreducible classes. Characteristics of this approach is also discussed by comparing with the conventional one based on Polya's theorem.
Document Details
- Document Type
- Pub Defense Publication
- Publication Date
- Mar 26, 2021
- Source ID
- 10.1002/adts.202000301
Entities
People
- Genki I. Prayogo
- Keishu Utimula
- Kenta Hongo
- Kousuke Nakano
- Ryo Maezono
Organizations
- Air Force Office of Scientific Research
- Japan Advanced Institute of Science and Technology
- National Institute for Materials Science
- Toyota