What is a complete geometric interpretation of the eigendecomposition of matrices?

Question

For reference, eigendecomposition of a matrix $A$ $\in R^{n \times n}$ is defined as:

$A = P \Lambda P^{-1}$

where $P$ is a matrix whose columns are the eigenvectors of $A$, and $\Lambda$ is a diagonal matrix whose entries are the corresponding eigenvalues. This is only possible when $P$ is invertible, which is true when the eigenvectors of $A$ form a basis of $R^n$.

What is a complete geometric interpretation of the eigendecomposition of matrix $A$? Many interpretations that I see on the internet and in lectures narrow down on a special case where $A$ is symmetric. I understand that when $A$ is symmetric, its eigenvectors form an orthonormal basis of the space, so $P$ is an orthogonal matrix signifying rotations (maybe flips too). And in this special case, $P \Lambda P^{-1}$ can be interpreted as a transformation where:

1. You first rotate the space ($P^{-1}$).
2. Then, you scale the rotated space alone the axes ($\Lambda$).
3. Finally, you rotate it back ($P$).

But eigendecomposition is defined for any general square matrix $A \in R^{n \times n}$ whose eigenvectors form a basis of $R^n$. The matrix $P$ need not be an orthogonal matrix signifying rotation/flips. What is the geometric interpretation of eigendecomposition for this general case?

Answer 1

In a first step, group the eigenvalues by value.

Any group belongs to a subspace.

The choice of the basis of any of the subspaces is arbitrary, completeness and linear independece implied.

A nonempty kernel, generally speaking, shows, that the linear representation space of the map has been to large.

By an invertible basis transform and index resorting, the matrix can be transformed into an upper triangualar block matrix of a zero block on the diagonal acting on the kernel and a nonsingular matrix on the diagonal acting on the complement.

So concentrate to nonsingular matrices.

Only normal matrices, commuting with their adjoints, yield a basis of eigenvectors.

Commuting with the adjoint means in essence, that within the given scalar product representation space of the linear map A, A and A* have the same eigenbasis.

So they can be diagonalized with the same transformation.

If all eigenvalues are different, the procedure is trivial.

Two eigenvectors of different eigenvalues are orthogonal in the given hermitean scalar product

$$\left< u_\lambda, (\lambda -\mu)\ u_\mu \right> = \left< u_\lambda , \lambda \ u_\mu \right>- \left< u_\lambda , \mu\ u_\mu \right> = \left< \lambda^* \ u_\lambda , u_\mu \right> - \left< u_\lambda , \mu\ u_\mu \right>= \left< A^* \ u_\lambda , u_\mu \right> - \left< u_\lambda , A u_\mu \right> =0$$

$$ (\lambda -\mu) \left< u_\lambda,\ u_\mu \right> = 0$$

It follows, that the matrix of the eigenbasis as columns is an eigenvector of A in matrix form with a digonal matrix as the generalized eigenvalue

$$ A \cdot \left(\begin{array}{c} e_{\lambda_1}^T\\ \vdots \\ e_{\lambda_n}^T \end{array} \right) = \left(\begin{array}{ccc} e_{\lambda_1}, 0, \dots,\dots\\ \vdots,\vdots,\vdots,\vdots \\ 0, \dots, 0, e_{\lambda_n}\end{array} \right) \cdot \left(\begin{array}{c} e_{\lambda_1}^T\\ \vdots \\ e_{\lambda_n}^T \end{array} \right) $$

Since all eigenvectors are orthogonal the matrix of eigenvectors is invertible and diagonalizes $A$ and $A^*$ simultanuously.

If there are multiple eigenvalues, any basis of the corresponding subspaces can be used. That is an arbitrary choice, implying a Gram Schmidt orthogonalization procedure, or, more frequently applied, one choses of a naturally given complete system of pairwise commuting matrices, that split the subspaces of degeneration by their different eigenvalues in that subspaces.

Answer 2

There are basically two interpretations of matrices. Either as transformation of vectors or as a change of basis. Your description for symmetric matrices ("rotation of space") corresponds to the latter.

For a symmetric matrix $A$ the Eigenvalue decomposition yields rotations ($P$) and scalings ($\Lambda$) of an arbitrary vector $x$. For an arbitrary matrix, it consists of rotation-and-stretching and scalings.

Basis change

In terms of a basis change, we can interpret $A x = P \Lambda P^{-1}$ as follows:

1. $\hat x = P^{-1} x$: Express $x= \sum_i x_i e_i$ in the Eigenbasis {$p_i$}, i.e. find the $\hat x_i$ which yield $x=\sum_i \hat x_i p_i$. This can also be understood as a transformation of the underlying space. Since the Eigenbasis must not be orthonormal (e.g. $p_1=(1,0), p_2=(1,1)$ could be a valid basis), the underlying space cannot only be rotated, but must also be stretched in some directions.
2. $\tilde x = \Lambda \hat x$: Scaling along the $p_i$.
3. $Q \tilde x$: Transform back into the original basis.

Transformation

In the second interpretation the vectors themselves get transformed instead of the underlying space.

1. $\hat x = P^{-1} x$: Transform the vector $x = \sum_i x_i p_i$ by transforming its basis vectors, i.e. each basis vector gets rotated and stretched, $p_i \rightarrow e_i$.
2. $\Lambda \hat x$: Scale along $e_i$.
3. $Q \tilde x$: Transform the vector back.