Why is it true that $x^{T}Ax=\sum_{i=1}^{n}\sum_{j=1}^{n}A_{ij}x_{i}x_{j}$

Question

I am following some notes on applying derivatives to vectors/matrices, I found the following:

$$f(x)=x^{T}Ax=\sum_{i=1}^{n}\sum_{j=1}^{n}A_{ij}x_{i}x_{j}$$.

I don't understand why this is true, can someone explain how this is derived? (Supposed multiplication is well defined)

Answer 1

It is just matrix multiplication, where the result is a $1\times 1$ matrix, and this is regarded as a scalar,

Take $n=2$, so $\mathbf x=\begin{bmatrix}x_1\\x_2\end{bmatrix}$ and $\mathbf A=\begin{bmatrix}A_{11}&A_{12}\\A_{21}&A_{22}\end{bmatrix}$

Then $$\begin{align}\mathbf x^\top \mathbf A\mathbf x &=\begin{bmatrix}x_1&x_2\end{bmatrix}\begin{bmatrix}A_{11}&A_{12}\\A_{21}&A_{22}\end{bmatrix}\begin{bmatrix}x_1\\x_2\end{bmatrix}\\ &= \begin{bmatrix}x_1 & x_2\end{bmatrix}\begin{bmatrix} A_{11}x_1+A_{12}x_2\\A_{21}x_1+A_{22}x_2\end{bmatrix}\\&=\begin{matrix}x_1A_{11}x_1+x_1A_{12}x_2+x_2A_{21}x_1+x_2A_{22}x_2\end{matrix}\\&= \sum_{i=1}^2\sum_{j=1}^2 x_iA_{ij}x_j\end{align}$$

Answer 2

We are given that $x$ is a $n\times 1$ column vector, which can be viewed as a matrix whose $(i, 1)$ entry is $x_i$ . And $A$ is an $n\times n$ matrix whose $(i, j)$ entry is $A_{ij}$. Since $x^T$ is $1\times n$, the product $x^TAx$ must be $1\times 1$. By definition of matrix multiplication we can write this single element as $$(x^TAx)_{11}=\sum_{i=1}^n \sum_{j=1}^n (x^T)_{1i} (A)_{ij} (x)_{j1}\stackrel{(\ast)}=\sum_i\sum_j (x)_{i1}A_{ij}(x)_{j1}=\sum_i\sum_j x_i A_{ij}x_j\ . $$ Step $(\ast)$ uses the definition of transpose: $(B^T)_{ij}=B_{ji}$ for any matrix $B$.