Cluster Validity

Abstract

Section 14 reviews the validity problem. The main difficulty is this: complex algorithms stand squarely between the data for which substructure is hypothesized and the solutions they generate; hence it is all but impossible to transfer a theoretical null hypothesis about X to U ∈ M _fc which can be used to statistically substantiate or repudiate the validity of algorithmically suggested clusters. As a result, a number of scalar measures of partition fuzziness (which are interesting in their own right) have been used as heuristic validity indicants. Sections 15, 16, and 17 discuss three such measures: the Anderson iris data surfaces in S15 and S17. S18 contains several approaches aimed towards connecting a null hypothesis about X to U ∈ M _fc : this idea is currently being heavily studied, and S18 is transitory at best. Sections 19 and 20 discuss measures of hard cluster validity which have been related by their inventors to fuzzy algorithms in several ways. S19 contains a particularly interesting application to the design of interstellar navigational systems. S20 provides an additional insight into the geometric property that data which cluster well using FCM algorithm (A11.1) must have.