What are the use cases related to cluster analysis of different distance metrics?

Question

I'm trying to use different distance metrics like Euclidean, Manhattan, cosine, chebyshev among other distance metrics in my k-means algorithm to calculate distances between the data points and the centers. In what situation would one distance metric be more useful over the other in a clustering scenario? [Comparing all the above mentioned distance metrics]

Answer 1

You ask a great question. Deza & Deza's Dictionary of Distances catalogs hundreds of families of distance functions. And there are others that are not mentioned there but in other books such as Cichocki et al Nonnegative Matrix and Tensor Factorizations

Some observations:

• Typically there are distinct weighing functions within any given family of metrics.

• Not all metrics are applicable to all situations, so there are metrics used in data analysis that would not concern topologists, as well as composite metrics such as Hausdorff distance and Earth Movers Distance that apply primarily to 2 sets of objects rather than 2 objects.

• Even the Greeks 2k years ago were aware of several, possibly up to 10, distinct definitions of mean, including of course arithmetic, geometric and harmonic means. It is not until the early 20th century that Chisini related all the definitions of means to an invariance principle. (Medians and quantiles are even more tricky requiring a formulation in terms of constrained optimization, not just invariance).

• Results of data analysis tasks such as classification, clustering, regression and dimensionality reduction can sometimes be very sensitive to the choice of metric (data held fixed) and also can lead to completely different methodologies for solving the problem. For example, consider linear correlation in $L_2$ ("Least Squares" or Gauss's solution) versus $L_1$ (Least Absolute Deviations or Laplace's solution) loss functions. The $L_2$ metric penalizes outliers quadratically, whereas the $L_1$ metric penalizes outliers linearly, and there are examples where the sign of the linear regressor can flip (eg, from positive to negative or vice versa) just by switching from one to the other. (Also you can think of $L_2$ as being a differential method, where solutions are found by setting derivatives equal to zero, versus $L_1$ method which geometrically involves polyhedra.)

In summary, the choice of metric can certainly influence the results of even basic statistical data analysis tasks. It's a shame that statisticians understand very well that results are relative to such loss functionals, but applied scientists typically do not understand this and will compute and report an answer, as opposed to the answers.