closeness of a set of vectors

Question

Is there some measure that captures the "closeness" of a set of vectors? Say I have a matrix,

$$ A = \left[ \begin{matrix} 0.8 & 0.15 & 0.05 \\ 0.82 & 0.09 & 0.09 \\ 0.78 & 0.08 & 0.14 \\ \end{matrix} \right] $$

and another matrix $$ B = \left[ \begin{matrix} 0.5 & 0.2 & 0.3 \\ 0.01 & 0.6 & 0.39 \\ 0 & 0 & 1 \end{matrix} \right] $$

Based on this hypothetical measure of "closeness", the row vectors of A are closer to each other than are the rows of B

For any two vectors, I could get their cosine similarity.

Furthermore, if my matrix was square and had nonzero determinant, the determinant would in some sense capture the "closeness" of the rows, as rows that are very "close", or similar, would span a parallelpiped with smaller volume than would a matrix whose rows were very different

But what if I had a bunch of vectors (say, the rows of a 100 x 3 matrix)? How would I measure the collective ''closeness'' of the row vectors? I'm looking for some sort of measure like the determinant, where if the row vectors of a square matrix are close to one another, then the volume of the parallelpiped spanned by these vectors is smaller than if they are quite different - except I could use it for a matrix that is non-square.

Answer 1

Assuming that your matrix (of row vectors) contains more rows than columns, you can calculate $\sqrt{\det(A^T A)}$. In the case that $A$ is a square matrix, this reduces to $$ \sqrt{\det(A^T A)} = \sqrt{\det(A^T)\det(A)} = \sqrt{\det(A)^2} =|\det(A)| $$ Furthermore, we will only have $\sqrt{\det(A^T A)} = 0$ when the columns of $A$ are not linearly independent.

Answer 2

A standard way to measure how "close" two sets of vectors are is in terms of the distance and principal angles of subspaces which they span.

Let $X\in\mathbb{R}^{n\times p}$ and $Y\in\mathbb{R}^{n\times q}$ be matrices containing bases of two subspaces $\mathcal{X},\mathcal{Y}\subset\mathbb{R}^n$, respectively, such that $p=\dim\mathcal{X}\geq\dim\mathcal{Y}=q$ (assuming $q\geq 1$). (In your case, $X=A^T$ and $Y=B^T$). The principal angles $\{\theta_i\}_{i=1}^q$ (where $\theta_i\in[0,\pi/2]$) between $\mathcal{X}$ and $\mathcal{Y}$ are defined in a recursive way by $$ \cos\theta_i=x_i^Ty_i=\max_{\substack{x\in\mathcal{X}\setminus\{0\}\\x\perp x_1,\ldots,x_{i-1}}}\max_{\substack{y\in\mathcal{Y}\setminus\{0\}\\y\perp y_1,\ldots,y_{i-1}}}\frac{x^Ty}{\|x\|_2\|y\|_2},\quad i=1,\ldots,q. $$

The principal angles can be computed using the SVD. Let $X=Q_XR_X$ and $Y=Q_YR_Y$ be the (thin) QR factorisations of $X$ and $Y$, respectively, that is, the columns of $Q_X$ and $Q_Y$ contain orthonormal bases of $\mathcal{X}$ and $\mathcal{Y}$ in their columns. Now compute the SVD of $Q_X^TQ_Y$: $$ Q_X^TQ_Y=U\Sigma V^T, $$ such that $U$ and $V$ are orthogonal matrices and the singular values in $\Sigma=\mathrm{diag}(\sigma_1,\ldots,\sigma_q)$ are ordered in the non-increasing order. It can be shown that the cosines of the principal angles are precisely the singular values $\sigma_i$: $$ \cos\theta_i = \sigma_i,\quad i=1,\ldots,q. $$

Note that if $\mathcal{X}=\mathcal{Y}$, all singular values of $Q_X^TQ_Y$ are equal to one, that is, the principal angles between $\mathcal{X}$ and $\mathcal{Y}$ are zero. On the other hand, if $\mathcal{X}\perp\mathcal{Y}$, then all singular values of $Q_X^TQ_Y$ are equal to zero, that is, the principal angles between $\mathcal{X}$ and $\mathcal{Y}$ are equal to $\pi/2$.

If $P_{\mathcal{X}}$ and $P_{\mathcal{Y}}$ are orthogonal projections onto $\mathcal{X}$ and $\mathcal{Y}$, respectively, then a natural way to define the distance between $\mathcal{X}$ and $\mathcal{Y}$ is $$ \mathrm{dist}(\mathcal{X},\mathcal{Y})=\|P_{\mathcal{X}}-P_{\mathcal{Y}}\|_2. $$ If $p=q$, this distance can be expressed in terms of principal angles as $$ \mathrm{dist}(\mathcal{X},\mathcal{Y})=\sqrt{1-(\cos\theta_q)^2}. $$