Confusion regarding real inner-products and the spectral theorem for symmetric matrices

Question

Suppose $g:\mathbb{R}^{n}\times\mathbb{R}^{n}\rightarrow\mathbb{R}$ is a real inner-product and $\vec{v}_{1}, \ldots, \vec{v}_{n}$ are vectors such that $g(\vec{v}_{i}, \vec{v}_{j}) = \delta_{ij}$ for all $i, j$. In the standard basis, we can represent $g$ as a real, symmetric, positive-definite matrix $A$ so that $g(\vec{v}, \vec{w}) = \vec{v}\,^{T}\!A\vec{w}$.

By the spectral theorem, there are eigenvectors $\vec{p}_{1}, \ldots, \vec{p}_{n}$ with eigenvalues $\lambda_{1}, \ldots, \lambda_{n}$ such that $\vec{p}_{i}\cdot\vec{p}_{j} = \delta_{ij}$, and there is an orthogonal matrix $P$ such that $P^{T}AP = L$ where $L = \text{diag}(\lambda_{1}, \ldots, \lambda_{n})$. Now presumably, $P$ can be interpreted as the change of basis matrix from the standard basis $(\vec{e}_{1}, \ldots, \vec{e}_{n})$ to $(\vec{v}_{1}, \ldots, \vec{v}_{n})$. In this new basis, the matrix representation of $g$ is diagonalized, and so $\vec{v}_{1}, \ldots, \vec{v}_{n}$ are orthogonal.

Now my confusion here is that I can in principle construct an inner-product $g$ for which the vectors $\vec{v}_{1}, \ldots, \vec{v}_{n}$ are not orthogonal, yet the transformation $(\vec{e}_{1}, \ldots, \vec{e}_{n})\rightarrow (\vec{v}_{1}, \ldots, \vec{v}_{n})$ is done by an orthogonal matrix. To put it briefly, where is the mistake in my thinking regarding this?

Answer 1

Why should the vectors $v_i$ coincide with the eigenvectors $p_i$ of $A$?

More explicitly, consider an inner product such that $e_1$ and $e_1+e_2$ are an orthonormal basis. You can check that this corresponds to $x^tAy$ where $A=\left(\begin{array}{cc} 1 & -1\\ -1 & 2\end{array}\right)$. But neither $e_1$ nor $e_1+e_2$ are eigenvectors of this matrix.

Answer 2

Ok, I've realized a couple of mistakes. I'll leave this here in case anyone else has similar confusion regarding this. We need to distinguish between orthogonality according to the dot-product and orthogonality according to $g$. For the sake of clear notation, we'll write $e(\vec{v}, \vec{w}) := \vec{v}\cdot\vec{w}$, so $e$ will be the standard Euclidean inner-product. We should avoid confusing $g$-orthogonality with $e$-orthogonality.

Now looking back at my initial question, I'm not sure why I said,

Now presumably, $P$ can be interpreted as the change of basis matrix from the standard basis $(\vec{e}_{1}, \ldots, \vec{e}_{n})$ to $(\vec{v}_{1}, \ldots, \vec{v}_{n})$.

This part seems to be outright incorrect. Presumably, I identified $(\vec{v}_{1}, \ldots, v_{n})$ with $(\vec{p}_{1}, \ldots, \vec{p}_{n})$ up to rescaling and permutations, because I mistakenly thought that the fact that both $(\vec{p}_{1}, \ldots, \vec{p}_{n})$ and $(\vec{v}_{1}, \ldots, v_{n})$ are $g$-orthogonal means they are equal up to rescaling and permutations. Just because $(\vec{v}_{1}, \ldots, \vec{v}_{n})$ is $g$-orthogonal, it does not mean this is the only $g$-orthogonal basis, so it doesn't have to be equal to $(\vec{p}_{1}, \ldots, \vec{p}_{n})$.

Given a matrix $A$ representing $g$ in the standard basis, the spectral theorem guarantees $A$ has eigenvectors $\vec{p}_{1}, \ldots, \vec{p}_{n}$ with eigenvalues $\lambda_{1}, \ldots, \lambda_{n}$. These eigenvectors have a special meaning: these are the vectors that are both $g$-orthogonal and $e$-orthogonal. If we normalize them, then the matrix $P = (\vec{p}_{1} \;\cdots\; \vec{p}_{n})$ is an orthogonal matrix in the sense that it satisfies $P^{T}P = PP^{T} = I$, and at the same time $P$ allows you to transform to the basis in which the matrix of $g$ is diagonalized. This is okay because basis $(\vec{p}_{1}, \ldots, \vec{p}_{n})$ is simultaneously $g$-orthogonal and $e$-orthogonal.

Example. I'll prove an example with $n=2$. Suppose I construct a real inner-product in which the vectors $$ \vec{v}_{1} = \begin{pmatrix} 1 \\ 3 \end{pmatrix} \quad\text{ and }\quad \vec{v}_{2} = \begin{pmatrix} -1 \\ 1 \end{pmatrix} $$ are $g$-orthonormal but not $e$-orthonormal (or even $e$-orthogonal). By setting $\vec{v}\,^{T}_{i}g\,\vec{v}_{j} = \delta_{ij}$ and assuming $g$ is symmetric I can solve for entries of $g$ to get $$ g = \begin{pmatrix} \tfrac{5}{8} & -\tfrac{1}{8} \\ -\tfrac{1}{8} & \tfrac{1}{8} \end{pmatrix}. $$ Now to diagonalize $g$, I find eigenvectors $$ \vec{p}_{1} = \frac{1}{\sqrt{10 + 4\sqrt{5}}}\begin{pmatrix} -2-\sqrt{5} \\ 1 \end{pmatrix} \quad\text{ and }\quad \vec{p}_{2} = \frac{1}{\sqrt{10 - 4\sqrt{5}}}\begin{pmatrix} -2+\sqrt{5} \\ 1 \end{pmatrix} $$ with eigenvalues $\lambda_{1} = (3+\sqrt{5})/8$ and $\lambda_{2} = (3-\sqrt{5})/8$, respectively. By taking $P = (\vec{p}_{1} \;\; \vec{p}_{2})$, we indeed find $P^{T}P = I$ and $P^{T}gP = L$ where $L = \text{diag}(\lambda_{1}, \lambda_{2})$. The eigenvectors $\vec{p}_{1}, \vec{p}_{2}$ are not what we started out with, which was $\vec{v}_{1}, \vec{v}_{2}$, and it is only $\vec{p}_{1}, \vec{p}_{2}$ that are simultaneously $g$-orthogonal and $e$-orthogonal whereas $\vec{v}_{1}, \vec{v}_{2}$ are only $g$-orthogonal. Hence $P$ corresponds to a rotation+reflection in the 2D plane with respect to inner-product $e$, and it diagonalizes $g$ at the same time.

Notes:

• I thought I'd point out one more thing that I think led to me be confused. Given a basis $\mathcal{B}$ you can construct a inner-product $g$ such that $\mathcal{B}$ is $g$-orthonormal (as I did in the example), and such an inner-product is unique. Thus, $\mathcal{B}$ uniquely determines $g$. However, given $g$ you can't uniquely specify a $g$-orthonormal basis (because there's more than one). Thus, $g$ does not uniquely determine $\mathcal{B}$ (contrary to what I thought).
• The uniqueness condition of the previous bullet point implies any basis $\mathcal{B}$ is $g$-orthonormal with respect to only one inner-product $g$. However, a basis like $\mathcal{P} = (\vec{p}_{1}, \ldots, \vec{p}_{n})$ in the preceding text is both $e$-orthogonal and $g$-orthogonal. This doesn't contradict anything, because the uniqueness applies to ortho-normality rather than ortho-gonality. The basis $\mathcal{P}$ may be both $e$-orthogonal and $g$-orthogonal, but it is orthonormal in only one inner-product.