Why is it that if $f(x) = x^T A x$ then $\nabla f = \frac{1}{2} (A+A^T) x$

Question

I have that $$f(\vec{x},\vec{y}) = \vec{x}^T A \vec{x}$$

I have seen the result online that $$\nabla f = \frac{1}{2} (A+A^T) \vec{x}$$ yet I can't understand why this is the case. How do you get from one on the top to the bottom?

The thing I am actually trying to figure out is why we can go from (if $A$ is symmetric)

$$f(\vec{x},\vec{y}) = \begin{bmatrix} \vec{x} \\ \vec{y} \end{bmatrix}^T A \begin{bmatrix} \vec{x} \\ \vec{y} \end{bmatrix}$$ $$f(\vec{x},\vec{y}) = \begin{bmatrix} \vec{x} \\ \vec{y} \end{bmatrix}^T \frac{1}{2} (A+A^T) \begin{bmatrix} \vec{x} \\ \vec{y} \end{bmatrix}$$

To

$$\nabla f = \frac{1}{2}(A+A^T) \begin{bmatrix} \vec{x} \\ \vec{y} \end{bmatrix}$$

Then apparently the Hessian is as follows:

$$H = \frac{1}{2} (A + A^T) = A$$

Could someone help me to understand?

Answer 1

Let $A=\left[\begin{array}{} a & b\\ c& d\end{array}\right]$, then $$ f(x,y)=ax^2+(b+c)xy+dy^2 $$ $$ \nabla f=\left[\begin{array}{} 2ax+(b+c)y \\ (b+c)x+2dy\end{array}\right]= \left[\begin{array}{} 2a & b+c \\ b+c & 2d\end{array}\right]\cdot \left[\begin{array}{} x \\ y\end{array}\right]= (A+A^T)\cdot \left[\begin{array}{} x \\ y\end{array}\right]. $$ Why there is $\frac12$ in your result?

Answer 2

We want to compute the gradient with finesse, and that means we should think in terms of Newton's approximation: $$ \tag{1} f(x + \Delta x) \approx f(x) + f'(x) \Delta x. $$ If $f(x) = x^T Ax$, and $\Delta x$ is a small vector, then \begin{align} f(x + \Delta x) &= (x + \Delta x)^T A (x + \Delta x) \\ &= x^T A x + x^T A \Delta x + \Delta x^T A x + \Delta x^T A \Delta x \\ &= x^T A x + x^T A \Delta x + x^T A^T \Delta x + \text{very small term} \\ &\approx x^T A x + x^T (A + A^T) \Delta x. \end{align} Comparing this with (1), we discover that $$ f'(x) = x^T(A + A^T). $$ So, $$ \nabla f(x) = f'(x)^T = (A + A^T) x. $$

Answer 3

Assuming you meant $f(x) = x^T A x$, notice that for any vector $h$, we have:

\begin{align*} f(x + h) &= (x + h)^T A (x + h) \\ &= x^T A x + h^T A x + x^T A h + h^T A h \\ &= f(x) + h^T (A + A^T)x + h^T A h \end{align*}

With $L(h) = h^T(A + A^T)x$, which is clearly linear, we find that

\begin{align*} \lim_{h \to 0} \frac{f(x + h) - f(x) - L(h)}{||h||} &= \lim_{h \to 0} \frac{h^T A h}{||h||} = 0 \end{align*}

So that $\nabla f(x) = (A + A^T)x$.

Answer 4

Young's theorem is that $$ \dfrac{\partial^2 f(x)}{\partial x_i \partial x_j} = \dfrac{\partial^2 f(x)}{\partial x_j \partial x_i} $$ so that cross derivatives of differentiable functions do not depend on the order of differentiation. $A$ is the Hessian of a quadratic form, so it must be symmetric. To make this really explicit, if you multiplied it out as $$ x'Ax = \sum_{i=1}^N \sum_{j=1}^N a_{ij} x_i x_j $$ and took the cross partial you would get $a_{ij} $ or $a_{ji}$ depending on the order in which you did it, so you must conclude $a_{ij} = a_{ji}$ and $A$ is symmetric, or else $f(x) = x'Ax$ is violating Young's theorem.

Why does this matter?

Consider a vector field $F(x)$ and any piecewise smooth curve from $x$ to $x'$, so the line integral is $$ \int_{0}^{1} F(\alpha(t)) d \alpha(t) $$ where $\alpha(0) = x$ and $\alpha(1) = x'$. We might ask, ``does the vector field $F(x)$ characterize a proper function, so $F(x) = \nabla f(x)$?'' If so, we'll need it to be the case that $$ f(x) - f(x') = \int_{0}^{1} F(\alpha(t)) d \alpha(t) $$ for any piecewise smooth curve $\alpha$. Basically, does the fundamental theorem of calculus hold?

If $A \neq A'$, however, $2A'x$ cannot be the gradient of a function, because it will have curl and violate Young's theorem (sometimes people use $\frac{1}{2}(A+A')$ to ``symmetricize'' $A$). So a necessary condition for $2A'x$ to be the gradient of a function is that $A$ be symmetric. Otherwise, the vector field $A'x$ would not be conservative, and line integrals would depend on the path, so they could not be integrated to get a potential function that makes sense of the vector field. The answer is not that $A$ is not symmetric and the Hessian is $\frac{1}{2}(A'+A)$: it is that $A$ must be symmetric because the Hessian of $x'Ax$ is $A$, and if you wrote $A$ as non-symmetric, you might be lucky enough that the way you do the computations will correct it for you.