Why does it seem that eigendecomposition requires that the decomposed matrix be diagonal?

Question

The eigen-decomposition of positive semi-definite matrices always exists. Given such a matrix $\mathbf{A}$, then, we have

$$\mathbf{Av}=\lambda\mathbf{v}$$

for a given eigen value $\lambda$ and associated eigenvector $\mathbf{v}$, which can then be rearranged as

$$\mathbf{A}-\lambda\mathbf{I}=0$$

Here is where I get confused. It is quite possible that the off-diagonal terms of an arbitrary positive semi-definite matrix $\mathbf{A}$ are not zero, but this last equation implies that they must always be zero. What am I misunderstanding here?

Answer 1

That rearrangement is incorrect. The difference $A−λI$ is not 0; it is only 0 when acting on $v$.

There is, in general, no way to "divide" out $v$ from the both sides of the equation - $v$ is a vector, not a number, and generally does not have a multiplicative inverse.

Answer 2

Recall the following equivalence:

Let $M$ be a square matrix. Then the following statements are equivalent:

1. $M$ is singular.
2. There exists $v \neq 0$ such that $Mv = 0$.
3. $\det(M) = 0$.

Also recall that $\det(M) = 0$ does not imply $M = 0$. Applying these to $M = A - \lambda I$, we see from the equivalence that $\det(A - \lambda I) = 0$ but this does not guarantee that $A - \lambda I = 0$.

Answer 3

First off, if a matrix is positive semi-definite, then there exists an orthonormal basis of $V$ (the vector field we are in) consisting of eigenvectors of that matrix. So we can say that the matrix of that linear transformation with respect to this basis is diagonal. However, that does not mean every matrix of that linear transformation is diagonal, just with respect to a basis consisting of eigenvectors.

Also, note that $Av=\lambda v$ implies $(A-\lambda I)v=0.$ This does not mean $A-\lambda I=0.$ This is because $A-\lambda I$ is an operator/linear transformation that maps $v$ (an eigenvector of $\lambda$) to $0.$ That doesn't mean it maps any other vector to $0.$ Put simply, $(A-\lambda I)v=0$ is the same as saying $T(v)=0$ for some linear transformation $T,$ but that obviously does not mean $T$ is the zero map. It just says $v\in \ker{T}.$