I've been trying (without success) to graphically understand why, if $S$ is a symmetric matrix, and I've got two random vectors $\vec{a}$ and $\vec{b}$, then:
$$\vec{a}^TS\vec{b}=\vec{b}^TS\vec{a}$$
I know that I could algebraically prove the above by just taking the transpose of the entire thing, and since the transpose of a symmetric matrix is just the diagonal matrix itself, and the transpose of a number is just that number, then:
$$\vec{a}^TS\vec{b}=\vec{b}^TS^T\vec{a}=\vec{b}^TS\vec{a}$$
If we accept the computational definition of the dot product, proving this is even easier!
Let $a_0,a_1 ...$ be the components of $\vec{a}$ along the eigenvectors of the symmetric matrix, which is an ortho-scaling transformation. That is, it scales space in orthogonal directions, in the directions of its eigenvectors, but doesn't rotate them.
Additionally, let $b_0, b_1...$ be the components of $\vec{b}$ along the eigenvectors of $S$.
Finally, let $\lambda_0, \lambda_1 ...$ be the diagonal entries of $S$. Then...
$\vec{a}^TS\vec{b}=a_0\lambda_0b_0 + a_1\lambda_1b_1 + a_2\lambda_2b_2+...$
...while
$\vec{b}^TS\vec{a}=b_0\lambda_0a_0 + b_1\lambda_1a_1 + b_2\lambda_2a_2+...$
...which are obviously the exact same thing.
So, computationally, it works. But I'm trying to gain some graphical intuition.
Thinking of this graphically, $S$ is the symmetric matrix, scaling space in orthogonal directions by their eigenvalues.
The inner product of two vectors is the magnitude of the projection of one vector onto the other multiplied by the magnitude of the other.
Here's a graphical demonstration of what I"m saying. I made the eigenvectors of $S$ point in the cartesian directions, for simplicity. That is, in this case, $S$ is a diagonal matrix, so in the pictures, I've instead denoted it with a $D$.
The double bars around the vectors mean "the magnitude of", and the double arrows means "the projection of (the vector on the left of the double arrows) onto (the vector on the right of the double arrows)."
Can anyone intuitively see why the two would be equal? I can't, and its really bothering me!
The reason can essentially be seen through the spectral theorem. Recall first that a matrix $A$ is diagonalizable if it can be written as $A = SDS^{-1}$, where $D$ is diagonal and $S$ is invertible. What this decomposition "instructs" the basis vectors to do is to transform your current basis to the basis in the columns of $S$ (this is done by multiplying by $S^{-1}$ on your original basis). The matrix $A$ in this new basis looks diagonal, hence you multiply by $D$. Finally, you want to transform back to the original basis of your problem so you multiply by $S$.
In the particular case of symmetric matrices, we have a guarantee on diagonalizability by the spectral theorem:
If we want to visualize this geometrically, we think of $A$ as having a special basis where it looks diagonal, hence in that basis all that the matrix does is stretch, shrink, and possibly flip each basis direction. The matrix $P$ then just encodes how to transform to this basis (via $P^T$) and then transform back (via $P$).
If we consider the inner product $x^TAy$ for vectors $x,y$ in our vector space, we can decompose $A$ as
$$ x^TAy \;\; =\;\; x^TPDP^Ty \;\; =\;\; \left (P^Tx\right )^TD \left (P^Ty\right ) \;\; =\;\; \widetilde{x}^TD\widetilde{y}. $$
Therefore under the change of basis with $P^T$ we see that the vectors $\widetilde{x}$ and $\widetilde{y}$ just have their components multiplied and then scaled via the components of $D$. Therefore, under this change of basis, it shouldn't really matter which one you multiply first. It should be the same as writing $\widetilde{y}^TD\widetilde{x}$ and this is equivalent to $y^TAx$.