In Nei 1972 he describes what is now called Nei's standard genetic distance. He defines at as follows:
Consider two randomly mating diploid populations, $X$ and $Y$, in which multiple alleles aggregrate at a locus. Let $x_i$ and $y_i$ be the frequencies of the $i$th alleles in $X$ and $Y$, respectively. The probability of identity of two randomly chosen genes is $j_X = \sum x_i^2$ in population $X$, while it is $j_Y = \sum y_i^2$ in population $Y$. The probability of identity of a gene from $X$ and a gene from $Y$ is $j_{XY} = \sum x_iy_i$. [...] The normalized identity of genes between $X$ and $Y$ with respect to this locus is defined as $$I_j = \frac{j_{XY}}{\sqrt{j_Xj_Y}}.$$[...] The normalized identity of genes between $X$ and $Y$ with respect to all loci is defined as $$I = \frac{J_{XY}}{\sqrt{J_XJ_Y}},$$ where $J_X$, $J_Y$ and $J_{XY}$ are the arithmetic means of $j_X$, $j_Y$, and $j_{XY}$, respectively, over all loci, including monomorphic loci. [...] The genetic distance between $X$ and $Y$ is then defined as $$D = - \log_e I.$$
He states in this work that his function does not have the triangle inequality.
In the present measure of genetic distance, the triangle inequality, which is often assumed in numerical taxonomy (see, e.g., Levandowsky and Winter 1971), does not hold.
Nei also said, "Evolution does not occur so as to assure this property at least at the nucleotide level." in reference to the triangle inequality.
Not having the triangle inequality disqualifies Nei's score from being a metric. I am curious if any of the other properties of a metric fail to hold.
- Due to the product of probabilities and norms always being non-negative, and the negative log of a number on $[0,1]$, it seems clear that it has the non-negativity property.
- The symmetry property also holds because of the commutativity of scalar multiplication.
- It is clear that $D[X,X] = 0$, but does the stronger identity of indiscernibles hold? While identity of indiscernibles has a broader meaning, in this context of a metric I more narrowly mean $D[X,Y] = 0 \iff X = Y$.
(a) when the number of loci is exactly $1$, then yes, the identity definitely holds
(b) when the number of loci is greater than $1$, then I doubt it, but I don't have a proof/specific counterexample
(a) In this case $X = (x_1, \dots, x_i, \dots, x_I)$ and $Y= (y_1, \dots, y_i, \dots, y_I)$ where $I$ is the number of alleles for the single locus. Both of these are probability/frequency vectors, so non-negative entries summing to 1.
Then the Cauchy-Schwarz inequality says that the dot product $\sum_{i=1}^I x_i y_i$ equals the product of the norms $\sqrt{\sum_{i=1^I} x_i^2} \cdot \sqrt{\sum_{i=1^I} y_i^2}$, i.e. their ratio equals 1, i.e. the negative logarithm of their ratio (Nei distance) equals $0$, if and only if the two vectors are collinear (scalar multiples of each other).
However, two non-negative vectors whose entries are both constrained to the same value can be scalar multiples of each other if and only if they are equal.
(b) Let $L$ equal the number of loci, and for each $\ell = 1, \dots, L$, let $I_{\ell}$ denote the number of alleles for the $\ell$'th locus. Then $$X = ( (x^{(1)}_1, \dots, x^{(1)}_{i_1}, \dots, x^{(1)}_{I_1}), \dots,(x^{(\ell)}_1, \dots, x^{(\ell)}_{i_\ell}, \dots, x^{(\ell)}_{I_\ell}) , \dots, (x^{(L)}_1, \dots, x^{(L)}_{i_L}, \dots, x^{(L)}_{I_L})) $$ $$Y = ( (y^{(1)}_1, \dots, y^{(1)}_{i_1}, \dots, y^{(1)}_{I_1}), \dots,(y^{(\ell)}_1, \dots, y^{(\ell)}_{i_\ell}, \dots, y^{(\ell)}_{I_\ell}) , \dots, (y^{(L)}_1, \dots, y^{(L)}_{i_L}, \dots, y^{(L)}_{I_L})) \,, $$ where each vector in the vector of vectors is a probability vector.
So the identity of indiscernibles should probably fail for more general reasons here. I think it should not be too hard to find a counterexample that bites you because of (i) symmetry on a per-locus basis, (ii) that the definitions of the $J_X$, $J_Y$, $J_{XY}$ are arithmetic means over the loci (and thus symmetric over the loci).
As an example of this, let $L=2$, with $I_1 = I_2 = 2$. So $$X = ((x_1^{(1)}, x_2^{(1)})), (x_1^{(2)}, x_2^{(2)}) \,, \quad Y = ((y_1^{(1)}, y_2^{(1)}), (y_1^{(2)}, y_2^{(2)})) \,. $$ (Also, yes, for the record, it would probably make sense to choose some way to structure these as a left or right stochastic matrix. However the number of alleles can differ between loci.) So then $$I = \frac{ \displaystyle\frac{1}{2}\left( \sum_{i_1=1}^2 x_{i_1}^{(1)}y_{i_1}^{(1)} + \sum_{i_2=1}^2 x_{i_2}^{(2)}y_{i_2}^{(2)} \right) }{ \displaystyle\sqrt{\frac{1}{2}\left( \sum_{i_1=1}^2 (x_{i_1}^{(1)})^2 + \sum_{i_2=1}^2 (x_{i_2}^{(2)})^2 \right)}\sqrt{\frac{1}{2} \left(\sum_{i_1=1}^2 (y_{i_1}^{(1)})^2 + \sum_{i_2=1}^2 (y_{i_2}^{(2)})^2 \right)}} \,. $$ So basically my proposed counterexample then, as mentioned above, suggests playing off the two kinds of symmetry against each other. Specifically, consider the special case when $$(x_1^{(1)}, x_2^{(1)}) = (y_1^{(2)}, y_2^{(2)}) \quad \text{and} \quad (x_1^{(2)}, x_2^{(2)}) = (y_1^{(1)}, y_2^{(1)}) \,.$$ Substituting all $y$ terms in the above formula with $x$ terms, we get $$\begin{array}{rcl}I & = & \frac{ \displaystyle\frac{1}{2}\left( \sum_{i_1=i_2=1}^2 x_{i_1}^{(1)}x_{i_2}^{(2)} + \sum_{i_1=i_2=1}^2 x_{i_2}^{(2)}x_{i_1}^{(1)} \right) }{ \displaystyle\sqrt{\frac{1}{2}\left( \sum_{i_1=i_2=1}^2 (x_{i_1}^{(1)})^2 + \sum_{i_1=i_2=1}^2 (x_{i_2}^{(2)})^2 \right)}\sqrt{\frac{1}{2} \left(\sum_{i_1=i_2=1}^2 (x_{i_2}^{(2)})^2 + \sum_{i_1=i_2=1}^2 (x_{i_1}^{(1)})^2 \right)}} \\ & = & \frac{ \displaystyle \sum_{i_1=i_2=1}^2 x_{i_1}^{(1)}x_{i_2}^{(2)} }{ \displaystyle \frac{1}{2}\left( \sum_{i_1=i_2=1}^2 (x_{i_1}^{(1)})^2 + \sum_{i_1=i_2=1}^2 (x_{i_2}^{(2)})^2 \right)} \,. \end{array}\tag{special}$$ Unfortunately this can never be a counterexample, because we are asking for two vector $\vec{x}^{(1)}=:v$ and $\vec{x}^{(2)}=:w$ such that $2 v \bullet w = ||v||^2 + ||w||^2$, which is true if and only if $(v_1^2 - 2v_1w_1 + w_1^2) + (v_2^2 - 2v_2 w_2 + w_2^2) = (v_1 - w_1)^2 + (v_2 - w_2)^2 = 0$. And that's only possible when $v_1 = w_1$ and $v_2 = w_2$, i.e. when $v = w$.
In any case, if there is any counterexample, it should probably be possible to find with $L=2=I_1 = I_2$. But I haven't been able to show that.