Second order derivative of $f(x):=\frac{1}{2} ⟨x,Ax⟩$

Question

Let $A=\left(A_{i j}\right)$ be an $n \times n$ symmetric matrix, and define the function $f: \mathbb{R}^n \rightarrow \mathbb{R}$ as $$ f(x):= \frac{1}{2} ​ ⟨x,Ax⟩ $$ Using the definition, determine the second-order derivative $D^2 f(a) \in \operatorname{Hom}^2\left(\mathbb{R}^n, \mathbb{R}\right)$. (Here, $\operatorname{Hom}^2\left(\mathbb{R}^n, \mathbb{R}\right)$ refers to the space of bilinear maps $\mathbb{R}^n \times \mathbb{R}^n \rightarrow \mathbb{R}$.).

I have found the derivative using the definition, that is $a^TA$, and I know that the definition of the second-order derivative is $\left(D^2 f\right)(a):=D(D f)(a) \in \operatorname{Hom}^2\left(\mathbb{R}^n, \mathbb{R}^m\right)$, but I don't know how to use this to find the second-order derivative.

Answer 1

Using the definition of inner product:

$$\langle x, Ax\rangle=\sum_{j}\sum_{i}x_j A_{i,j} x_i $$

Taking the $\partial_{x_k}$ component to the inner product:

$${\partial \over \partial {x_k}}\sum_{j}\sum_{i}x_j A_{i,j} x_i=\sum_{j}\sum_{i}{\partial \over \partial {x_k}}x_j A_{i,j} x_i$$

$$=2A_{k,k}x_k+\sum_k\sum_{i\neq k}A_{i,k}x_i$$

Then for the second "derivative" (in reality the Jacobian of the gradient, the Hessian matrix), has component of mixed partial derivatives, so you need to take partial derivative of one component with respect of all components.

$${\partial \over \partial x_l}{\partial \over \partial x_k}\langle x, Ax\rangle={\partial \over \partial x_l}\left(2A_{k,k}x_k+\sum_k\sum_{i\neq k}A_{i,k}x_i\right)$$

$$= \begin{cases} 2 A_{k,k} & k=l \\ \sum_k\sum_{i\neq k}A_{i,k} & k\neq l \end{cases} $$

$${d^2\over d\mathbf x^2}\langle x,Ax\rangle=A+A^T $$

In your case, is $2A$ because $A$ is symmetric and the last result is just $A$ for the factor of $1/2$.

EDIT:

If you use the Einstein summation convention, its more intuitive:

$$\langle x, Ax\rangle=A_j^i x_i x^j$$ $$\Rightarrow {\partial \over \partial x_k}\left(A_j^i x^j x_i \right)=A_j^i\delta_k^j x_i+A_j^i x^j \delta^k_i=A_k^i x_i+A_j^k x^j$$ $$\Rightarrow {d\over d\mathbf x}\langle x, Ax\rangle=x^T(A+A^T)$$

$$ $$

$${\partial \over \partial x_l}\left(A_k^i x_i+A_j^k x^j \right)=A_k^i \delta_i^l+A_j^k \delta^j_l=A_k^l+A_l^k $$ $$\Rightarrow{d\over d\mathbf x}x^T(A+A^T)=A+A^T $$

$$ $$

$${d^2\over d\mathbf x^2}\langle x, Ax\rangle=A+A^T$$

Answer 2

Because you wrote $f(x) = \frac12\langle x, Ax\rangle$ I'm going to keep using $x$ consistently and write $$ Df(x)(u) = \langle x, Au\rangle = x^TAu. $$ We want to differentiate this expression again with respect to $x$; but it is a linear function in $x$, and the derivative of a linear function is that function itself. Hence $$ D^2f(x)(u, v) = D[x \mapsto Df(x)(u)](v) = \langle v, Au\rangle, $$ or in matrix notation $$ D^2f(x)(u, v) = v^TAu. $$

Answer 3

I did it in a different way, where $C$ is the second-order derivative, $$ \begin{aligned} r'(h) &= f'(a+h) - f'(a) - Ch \\ &= (a+h)^T A - a^TA - Ch \\ &= a^TA + h^T - a^T A - Ch \\ &= h^TA - Ch \end{aligned}. $$ and now take the limit of $h$ to $0$, $$ \begin{aligned} \lim_{h \to 0} \frac{h^TA - Ch}{||h||} = \lim_{h \to 0} \frac{h^TA - Ch}{h} = A - C. \end{aligned} $$ and choose $C = A$ for the term to be equal to $0$ and so $A$ is the second-order derivative.