How Many Clusters: A Validation Index for Arbitrary-Shaped Clusters - Baya, Ariel Emilio (2013)
Clustering validation indexes are intended to assess the goodness of clustering results. Many methods used to estimate the number of clusters rely on a validation index as a key element to ﬁnd the correct answer. This paper presents a new validation index based on graph concepts, which has been designed to ﬁnd arbitrary shaped clusters by exploiting the spatial layout of the patterns and their clustering label. This new clustering index is combined with a solid statistical detection framework, the Gap Statistic. The resulting method is able to ﬁnd the right number of arbitrary shaped clusters in diverse situations, as we show with examples where this information is available. A comparison with several relevant validation methods is carried out using artiﬁcial and gene expression datasets. The results are very encouraging, showing that the underlying structure in the data can be more accurately detected with the new clustering index. Our gene expression data results also indicate that this new index is stable under perturbation of the input data.