Hierarchical Clustering
In data mining and statistics, hierarchical clustering (also called hierarchical cluster analysis or HCA) is a method of cluster analysis which seeks to build a hierarchy of clusters. Strategies for hierarchical clustering generally fall into two types:
Type | Description |
---|---|
Agglomerative | This is a bottom-up approach: each observation starts in its own cluster, and pairs of clusters are merged as one moves up the hierarchy |
Divisive | This is a top-down approach: all observations start in one cluster, and splits are performed recursively as one moves down the hierarchy |
In general, the merges and splits are determined in a greedy manner. The results of hierarchical clustering^{[1]} are usually presented in a dendrogram.
Cluster dissimilarity
In order to decide which clusters should be combined (for agglomerative), or where a cluster should be split (for divisive), a measure of dissimilarity between sets of observations is required. In most methods of hierarchical clustering, this is achieved by use of an appropriate metric (a measure of distance between pairs of observations), and a linkage criterion which specifies the dissimilarity of sets as a function of the pairwise distances of observations in the sets.
Metric
The choice of an appropriate metric will influence the shape of the clusters, as some elements may be relatively closer to one another under one metric than another. For example, in two dimensions, under the Manhattan distance metric, the distance between the origin (0,0) and (0.5, 0.5) is the same as the distance between the origin and (0, 1), while under the Euclidean distance metric the latter is strictly greater.
Some commonly used metrics for hierarchical clustering are:^{[2]}
Names | Formula |
---|---|
Euclidean distance | [math] \|a-b \|_2 = \sqrt{\sum_i (a_i-b_i)^2} [/math] |
Squared Euclidean distance | [math] \|a-b \|_2^2 = \sum_i (a_i-b_i)^2 [/math] |
Manhattan (or city block ) distance | [math] \|a-b \|_1 = \sum_i |a_i-b_i| [/math] |
Maximum distance (or Chebyshev distance) | [math] \|a-b \|_\infty = \max_i |a_i-b_i| [/math] |
Mahalanobis distance | [math] \sqrt{(a-b)^{\top}S^{-1}(a-b)} [/math] where S is the Covariance matrix |
For text or other non-numeric data, metrics such as the Hamming distance or Levenshtein distance are often used.
Euclidean and Manhattan distances are the special cases of generalized Minkowski distance with p = 1 (for Manhattan) and p = 2 (for Euclidean).
Several other dissimilarity measures exist. Particularly, correlation-based distances - Pearson, Eisen cosine, Spearman, Kendall correlation distances, which are widely used for gene expression data analyses. Correlation-based distance is defined by subtracting the correlation coefficient from 1. Strictly speaking, correlation-based distances cannot be used as metric, while the square root of it can be.^{[3]}
A review of cluster analysis in health psychology research found that the most common distance measure in published studies in that research area is the Euclidean distance or the squared Euclidean distance.
Linkage criteria
The linkage criterion determines the distance between sets of observations as a function of the pairwise distances between observations.
Some commonly used linkage criteria between two sets of observations [math]A[/math] and [math]B[/math] are:^{[4]}^{[5]}
Names | Formula |
---|---|
Maximum or complete-linkage clustering | [math] \max \, \{\, d(a,b) : a \in A,\, b \in B \,\}. [/math] |
Minimum or single-linkage clustering | [math] \min \, \{\, d(a,b) : a \in A,\, b \in B \,\}.. [/math] |
Unweighted average linkage clustering (or UPGMA) | [math] \frac{1}{|A|\cdot|B|} \sum_{a \in A }\sum_{ b \in B} d(a,b). [/math] |
Weighted average linkage clustering (or WPGMA) | [math] d(i \cup j, k) = \frac{d(i, k) + d(j, k)}{2}. [/math] |
Centroid linkage clustering, or UPGMC | [math] d(c_A,c_B) [/math] where [math]c_A[/math] and [math]c_B[/math] are the centroids of clusters [math]A[/math] and [math]B[/math], respectively. |
Minimum energy clustering | [math] \frac {2}{nm}\sum_{i,j=1}^{n,m} \|a_i- b_j\|_2 - \frac {1}{n^2}\sum_{i,j=1}^{n} \|a_i-a_j\|_2 - \frac{1}{m^2}\sum_{i,j=1}^{m} \|b_i-b_j\|_2 [/math] |
Complete Linkage Clustering
Complete-linkage clustering is one of several methods of agglomerative hierarchical clustering. At the beginning of the process, each element is in a cluster of its own. The clusters are then sequentially combined into larger clusters until all elements end up being in the same cluster. The method is also known as farthest neighbour clustering. The result of the clustering can be visualized as a dendrogram, which shows the sequence of cluster fusion and the distance at which each fusion took place.^{[6]}^{[7]}^{[8]}
Clustering procedure
At each step, the two clusters separated by the shortest distance are combined. The definition of 'shortest distance' is what differentiates between the different agglomerative clustering methods. In complete-linkage clustering, the link between two clusters contains all element pairs, and the distance between clusters equals the distance between those two elements (one in each cluster) that are farthest away from each other. The shortest of these links that remains at any step causes the fusion of the two clusters whose elements are involved.
Mathematically, the complete linkage function — the distance [math]D(X,Y)[/math] between clusters [math]X[/math] and [math]Y[/math] — is described by the following expression :
where [math]d(x,y)[/math] is the distance between elements [math]x \in X[/math] and [math]y \in Y[/math], and [math]X[/math] and [math]Y[/math] are two sets of elements (clusters).
Algorithms
Naive scheme
The following algorithm is an agglomerative scheme that erases rows and columns in a proximity matrix as old clusters are merged into new ones. The [math]N \times N[/math] proximity matrix [math]D[/math] contains all distances [math]d(i,j)[/math]. The clusterings are assigned sequence numbers [math]0,1,\ldots,n-1[/math] and [math]L(k)[/math] is the level of the [math]k[/math]-th clustering. A cluster with sequence number [math]m[/math] is denoted ([math]m[/math]) and the proximity between clusters ([math]r[/math]) and ([math]s[/math]) is denoted [math]d[(r),(s)][/math].
The complete linkage clustering algorithm consists of the following steps:
- Begin with the disjoint clustering having level [math]L(0) = 0[/math] and sequence number [math]m=0[/math].
- Find the most similar pair of clusters in the current clustering, say pair [math](r), (s)[/math], according to [math]d[(r),(s)] = \min d[(i),(j)][/math]where the minimum is over all pairs of clusters in the current clustering.
- Increment the sequence number: [math]m = m + 1[/math]. Merge clusters [math](r)[/math] and [math](s)[/math] into a single cluster to form the next clustering [math]m[/math]. Set the level of this clustering to [math]L(m) = d[(r),(s)][/math]
- Update the proximity matrix, [math]D[/math], by deleting the rows and columns corresponding to clusters [math](r)[/math] and [math](s)[/math] and adding a row and column corresponding to the newly formed cluster. The proximity between the new cluster, denoted [math](r,s)[/math] and old cluster [math](k)[/math] is defined as [math]d[(r), (s)] = \max \{d[(k),(r)], d[(k),(s)] \}[/math].
- If all objects are in one cluster, stop. Else, go to step 2.
Working example
The working example is based on a JC69 genetic distance matrix computed from the 5S ribosomal RNA sequence alignment of five bacteria: Bacillus subtilis ([math]a[/math]), Bacillus stearothermophilus ([math]b[/math]), Lactobacillus viridescens ([math]c[/math]), Acholeplasma modicum ([math]d[/math]), and Micrococcus luteus ([math]e[/math]).^{[9]}^{[10]}
First step
First clustering
Let us assume that we have five elements [math](a,b,c,d,e)[/math] and the following matrix [math]D_1[/math] of pairwise distances between them:
a | b | c | d | e | |
---|---|---|---|---|---|
a | 0 | 17 | 21 | 31 | 23 |
b | 17 | 0 | 30 | 34 | 21 |
c | 21 | 30 | 0 | 28 | 39 |
d | 31 | 34 | 28 | 0 | 43 |
e | 23 | 21 | 39 | 43 | 0 |
In this example, [math]D_1 (a,b)=17[/math] is the smallest value of [math]D_1[/math], so we join elements [math]a[/math] and [math]b[/math].
First branch length estimation
Let [math]u[/math] denote the node to which [math]a[/math] and [math]b[/math] are now connected. Setting [math]\delta(a,u)=\delta(b,u)=D_1(a,b)/2[/math] ensures that elements [math]a[/math] and [math]b[/math] are equidistant from [math]u[/math]. This corresponds to the expectation of the ultrametricity hypothesis. The branches joining [math]a[/math] and [math]b[/math] to [math]u[/math] then have lengths [math]\delta(a,u)=\delta(b,u)=17/2=8.5[/math]
First distance matrix update
We then proceed to update the initial proximity matrix [math]D_1[/math] into a new proximity matrix [math]D_2[/math] (see below), reduced in size by one row and one column because of the clustering of [math]a[/math] with [math]b[/math]. Bold values in [math]D_2[/math] correspond to the new distances, calculated by retaining the maximum distance between each element of the first cluster [math](a,b)[/math] and each of the remaining elements:
[math]D_2((a,b),c)=max(D_1(a,c),D_1(b,c))=max(21,30)=30[/math]
[math]D_2((a,b),d)=max(D_1(a,d),D_1(b,d))=max(31,34)=34[/math]
[math]D_2((a,b),e)=max(D_1(a,e),D_1(b,e))=max(23,21)=23[/math]
Italicized values in [math]D_2[/math] are not affected by the matrix update as they correspond to distances between elements not involved in the first cluster.
Second step
Second clustering
We now reiterate the three previous steps, starting from the new distance matrix [math]D_2[/math] :
(a,b) | c | d | e | |
---|---|---|---|---|
(a,b) | 0 | 30 | 34 | 23 |
c | 30 | 0 | 28 | 39 |
d | 34 | 28 | 0 | 43 |
e | 23 | 39 | 43 | 0 |
Here, [math]D_2 ((a,b),e)=23[/math] is the lowest value of [math]D_2[/math], so we join cluster [math](a,b)[/math] with element [math]e[/math].
- Second branch length estimation
Let [math]v[/math] denote the node to which [math](a,b)[/math] and [math]e[/math] are now connected. Because of the ultrametricity constraint, the branches joining [math]a[/math] or [math]b[/math] to [math]v[/math], and [math]e[/math] to [math]v[/math], are equal and have the following total length: [math]\delta(a,v)=\delta(b,v)=\delta(e,v)=23/2=11.5[/math]
We deduce the missing branch length: [math]\delta(u,v)=\delta(e,v)-\delta(a,u)=\delta(e,v)-\delta(b,u)=11.5-8.5=3[/math]
Second distance matrix update
We then proceed to update the [math]D_2[/math] matrix into a new distance matrix [math]D_3[/math] (see below), reduced in size by one row and one column because of the clustering of [math](a,b)[/math] with [math]e[/math] :
[math]D_3(((a,b),e),c)=max(D_2((a,b),c),D_2(e,c))=max(30,39)=39[/math]
[math]D_3(((a,b),e),d)=max(D_2((a,b),d),D_2(e,d))=max(34,43)=43[/math]
Third step
Third clustering
We again reiterate the three previous steps, starting from the updated distance matrix [math]D_3[/math].
((a,b),e) | c | d | |
---|---|---|---|
((a,b),e) | 0 | 39 | 43 |
c | 39 | 0 | 28 |
d | 43 | 28 | 0 |
Here, [math]D_3 (c,d)=28[/math] is the smallest value of [math]D_3[/math], so we join elements [math]c[/math] and [math]d[/math].
Third branch length estimation
Let [math]w[/math] denote the node to which [math]c[/math] and [math]d[/math] are now connected. The branches joining [math]c[/math] and [math]d[/math] to [math]w[/math] then have lengths [math]\delta(c,w)=\delta(d,w)=28/2=14[/math] (see the final dendrogram)
Third distance matrix update
There is a single entry to update: [math]D_4((c,d),((a,b),e))=max(D_3(c,((a,b),e)), D_3(d,((a,b),e)))=max(39, 43)=43[/math]
Final step
The final [math]D_4[/math] matrix is:
((a,b),e) | (c,d) | |
---|---|---|
((a,b),e) | 0 | 43 |
(c,d) | 43 | 0 |
So we join clusters [math]((a,b),e)[/math] and [math](c,d)[/math]. Let [math]r[/math] denote the (root) node to which [math]((a,b),e)[/math] and [math](c,d)[/math] are now connected. The branches joining [math]((a,b),e)[/math] and [math](c,d)[/math] to [math]r[/math] then have lengths:
[math]\delta(((a,b),e),r)=\delta((c,d),r)=43/2=21.5[/math]
We deduce the two remaining branch lengths:
[math]\delta(v,r)=\delta(((a,b),e),r)-\delta(e,v)=21.5-11.5=10[/math]
[math]\delta(w,r)=\delta((c,d),r)-\delta(c,w)=21.5-14=7.5[/math]
The complete-linkage dendrogram
The dendrogram is now complete. It is ultrametric because all tips ([math]a[/math] to [math]e[/math]) are equidistant from [math]r[/math] :
[math]\delta(a,r)=\delta(b,r)=\delta(e,r)=\delta(c,r)=\delta(d,r)=21.5[/math]
The dendrogram is therefore rooted by [math]r[/math], its deepest node.
Single-linkage Clustering
In statistics, single-linkage clustering is one of several methods of hierarchical clustering. It is based on grouping clusters in bottom-up fashion (agglomerative clustering), at each step combining two clusters that contain the closest pair of elements not yet belonging to the same cluster as each other.
A drawback of this method is that it tends to produce long thin clusters in which nearby elements of the same cluster have small distances, but elements at opposite ends of a cluster may be much farther from each other than two elements of other clusters. This may lead to difficulties in defining classes that could usefully subdivide the data.^{[11]}
Overview of agglomerative clustering methods
In the beginning of the agglomerative clustering process, each element is in a cluster of its own. The clusters are then sequentially combined into larger clusters, until all elements end up being in the same cluster. At each step, the two clusters separated by the shortest distance are combined. The function used to determine the distance between two clusters, known as the linkage function, is what differentiates the agglomerative clustering methods.
In single-linkage clustering, the distance between two clusters is determined by a single pair of elements: those two elements (one in each cluster) that are closest to each other. The shortest of these pairwise distances that remain at any step causes the two clusters whose elements are involved to be merged. The method is also known as nearest neighbour clustering. The result of the clustering can be visualized as a dendrogram, which shows the sequence in which clusters were merged and the distance at which each merge took place.^{[12]}
Mathematically, the linkage function – the distance [math]D(X,Y)[/math] between clusters [math]X[/math] and [math]Y[/math]– is described by the expression
where [math]X[/math] and [math]Y[/math] are any two sets of elements considered as clusters, and [math]d(x,y)[/math] denotes the distance between the two elements [math]x[/math] and [math]y[/math].
Naive algorithm
The following algorithm is an agglomerative scheme that erases rows and columns in a proximity matrix as old clusters are merged into new ones. The [math]N \times N[/math] proximity matrix [math]D[/math] contains all distances [math]d(i,j)[/math]. The clusterings are assigned sequence numbers [math]0,1, \ldots, n-1[/math] and [math]L(k)[/math] is the level of the [math]k[/math]-th clustering. A cluster with sequence number [math]m[/math] is denoted ([math]m[/math]) and the proximity between clusters [math](r)[/math] and [math](s)[/math] is denoted [math]d[(r),(s)][/math].
The single linkage algorithm is composed of the following steps:
- Begin with the disjoint clustering having level [math]L(0) = 0[/math] and sequence number [math]m=0[/math].
- Find the most similar pair of clusters in the current clustering, say pair [math](r), (s)[/math], according to [math]d[(r),(s)] = \min d[(i),(j)][/math]where the minimum is over all pairs of clusters in the current clustering.
- Increment the sequence number: [math]m = m + 1[/math]. Merge clusters [math](r)[/math] and [math](s)[/math] into a single cluster to form the next clustering [math]m[/math]. Set the level of this clustering to [math]L(m) = d[(r),(s)][/math]
- Update the proximity matrix, [math]D[/math], by deleting the rows and columns corresponding to clusters [math](r)[/math] and [math](s)[/math] and adding a row and column corresponding to the newly formed cluster. The proximity between the new cluster, denoted [math](r,s)[/math] and old cluster [math](k)[/math] is defined as [math]d[(r,s),(k)] = \min \{d[(k),(r)], d[(k),(s)] \}[/math].
- If all objects are in one cluster, stop. Else, go to step 2.
Working example
This working example is based on a JC69 genetic distance matrix computed from the 5S ribosomal RNA sequence alignment of five bacteria: Bacillus subtilis ([math]a[/math]), Bacillus stearothermophilus ([math]b[/math]), Lactobacillus viridescens ([math]c[/math]), Acholeplasma modicum ([math]d[/math]), and Micrococcus luteus ([math]e[/math]).^{[9]}^{[10]}
First step
First clustering
Let us assume that we have five elements [math](a,b,c,d,e)[/math] and the following matrix [math]D_1[/math] of pairwise distances between them:
a | b | c | d | e | |
---|---|---|---|---|---|
a | 0 | 17 | 21 | 31 | 23 |
b | 17 | 0 | 30 | 34 | 21 |
c | 21 | 30 | 0 | 28 | 39 |
d | 31 | 34 | 28 | 0 | 43 |
e | 23 | 21 | 39 | 43 | 0 |
In this example, [math]D_1 (a,b)=17[/math] is the lowest value of [math]D_1[/math], so we cluster elements a and b.
First branch length estimation
Let u denote the node to which a and b are now connected. Setting [math]\delta(a,u)=\delta(b,u)=D_1(a,b)/2[/math] ensures that elements a and b are equidistant from u. This corresponds to the expectation of the ultrametricity hypothesis. The branches joining a and b to u then have lengths [math]\delta(a,u)=\delta(b,u)=17/2=8.5[/math] (see the final dendrogram)
First distance matrix update
We then proceed to update the initial proximity matrix [math]D_1[/math] into a new proximity matrix [math]D_2[/math] (see below), reduced in size by one row and one column because of the clustering of a with b. Bold values in [math]D_2[/math] correspond to the new distances, calculated by retaining the minimum distance between each element of the first cluster [math](a,b)[/math] and each of the remaining elements:
- [[math]]\begin{array}{lllllll} D_2((a,b),c)&=&\min(D_1(a,c),D_1(b,c))&=&\min(21,30)&=&21 \\ D_2((a,b),d)&=&\min(D_1(a,d),D_1(b,d))&=&\min(31,34)&=&31 \\ D_2((a,b),e)&=&\min(D_1(a,e),D_1(b,e))&=&\min(23,21)&=&21 \end{array}[[/math]]
Italicized values in [math]D_2[/math] are not affected by the matrix update as they correspond to distances between elements not involved in the first cluster.
Second step
Second clustering
We now reiterate the three previous actions, starting from the new distance matrix [math]D_2[/math] :
(a,b) | c | d | e | |
---|---|---|---|---|
(a,b) | 0 | 21 | 31 | 21 |
c | 21 | 0 | 28 | 39 |
d | 31 | 28 | 0 | 43 |
e | 21 | 39 | 43 | 0 |
Here, [math]D_2 ((a,b),c)=21[/math] and [math]D_2 ((a,b),e)=21[/math] are the lowest values of [math]D_2[/math], so we join cluster [math](a,b)[/math] with element c and with element e.
Second branch length estimation
Let v denote the node to which [math](a,b)[/math], c and e are now connected. Because of the ultrametricity constraint, the branches joining a or b to v, and c to v, and also e to v are equal and have the following total length:
We deduce the missing branch length:
Second distance matrix update
We then proceed to update the [math]D_2[/math] matrix into a new distance matrix [math]D_3[/math] (see below), reduced in size by two rows and two columns because of the clustering of [math](a,b)[/math] with c and with e :
Final step
The final [math]D_3[/math] matrix is:
((a,b),c,e) | d | |
---|---|---|
((a,b),c,e) | 0 | 28 |
d | 28 | 0 |
So we join clusters [math]((a,b),c,e)[/math] and [math]d[/math]. Let [math]r[/math] denote the (root) node to which [math]((a,b),c,e)[/math] and [math]d[/math] are now connected. The branches joining [math]((a,b),c,e)[/math] and [math]d[/math] to [math]r[/math] then have lengths:
[math]\delta(((a,b),c,e),r)=\delta(d,r)=28/2=14[/math]
We deduce the remaining branch length:
[math]\delta(v,r)=\delta(a,r)-\delta(a,v)=\delta(b,r)-\delta(b,v)=\delta(c,r)-\delta(c,v)=\delta(e,r)-\delta(e,v)=14-10.5=3.5[/math]
The single-linkage dendrogram
The dendrogram is now complete. It is ultrametric because all tips ([math]a[/math], [math]b[/math], [math]c[/math], [math]e[/math], and [math]d[/math]) are equidistant from [math]r[/math] :
[math]\delta(a,r)=\delta(b,r)=\delta(c,r)=\delta(e,r)=\delta(d,r)=14[/math]
The dendrogram is therefore rooted by [math]r[/math], its deepest node.
References
- Nielsen, Frank (2016). "8. Hierarchical Clustering". Introduction to HPC with MPI for Data Science. Springer. pp. 195–211. ISBN 978-3-319-21903-5.
- "The DISTANCE Procedure: Proximity Measures". SAS/STAT 9.2 Users Guide. SAS Institute. Retrieved 2009-04-26.
- Solo, Victor (2019). "Pearson Distance is not a Distance". arXiv:1908.06029 [stat.ME].
- "The CLUSTER Procedure: Clustering Methods". SAS/STAT 9.2 Users Guide. SAS Institute. Retrieved 2009-04-26.
- "Hierarchical clustering via Joint Between-Within Distances: Extending Ward's Minimum Variance Method" (2005). Journal of Classification 22 (2): 151–183. doi: .
- "A method of establishing groups of equal amplitude in plant sociology based on similarity of species and its application to analyses of the vegetation on Danish commons." (1948). Biologiske Skrifter 5: 1–34.
- Legendre P, Legendre L (1998). Numerical Ecology (Second English ed.). p. 853.
- Everitt BS, Landau S, Leese M (2001). Cluster Analysis (Fourth ed.). London: Arnold. ISBN 0-340-76119-9.
- ^{9.0} ^{9.1} "Collection of published 5S, 5.8S and 4.5S ribosomal RNA sequences" (1986). Nucleic Acids Research 14 Suppl (Suppl): r1-59. doi: . PMID 2422630.
- ^{10.0} ^{10.1} "Phylogenetic analysis using ribosomal RNA" (1988). Methods in Enzymology 164: 793–812. doi: . PMID 3241556.
- Everitt B (2011). Cluster analysis. Chichester, West Sussex, U.K: Wiley. ISBN 9780470749913.
- Legendre P, Legendre L (1998). Numerical Ecology. Developments in Environmental Modelling. 20 (Second English ed.). Amsterdam: Elsevier.
Wikipedia References
- Wikipedia contributors. "Hierarchical clustering". Wikipedia. Wikipedia. Retrieved 17 August 2022.
- Wikipedia contributors. "Single-linkage clustering". Wikipedia. Wikipedia. Retrieved 17 August 2022.
- Wikipedia contributors. "Complete-linkage clustering". Wikipedia. Wikipedia. Retrieved 17 August 2022.