Gradient and Hessian of $x x^T$ w.r.t. $x$, where $x \in \mathbb{R}^{n \times 1}$,?

Question

Question: Can we find the gradient and Hessian of $x x^T$ w.r.t. $x$, where $x \in \mathbb{R}^{n \times 1}$ ?

EDIT: If we can, may I know how to compute that? Thank you.

Answer 1

Gradient $$\frac{\partial \mathbf{Y}}{\partial x_i} = \begin{bmatrix} \frac{\partial y_{11}}{\partial x_i} & \frac{\partial y_{12}}{\partial x_i} & \cdots & \frac{\partial y_{1n}}{\partial x_i}\\ \frac{\partial y_{21}}{\partial x_i} & \frac{\partial y_{22}}{\partial x_i} & \cdots & \frac{\partial y_{2n}}{\partial x_i}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial y_{m1}}{\partial x_i} & \frac{\partial y_{m2}}{\partial x_i} & \cdots & \frac{\partial y_{mn}}{\partial x_i}\\ \end{bmatrix}.$$

Let $$\mathbf{Y} = \mathbf{xx^T} = \begin{bmatrix} x_1^2 & x_1x_2 & \ldots & x_1x_n \\ x_1x_2 & x_2^2 & \ldots & x_2x_n \\ \ldots &\ldots & \ldots & \ldots \\ x_nx_1 & x_nx_2 & \ldots & x_n^2 \\ \end{bmatrix}$$ So $$\frac{\partial \mathbf{xx^T}}{\partial x_i} = \mathbf{Z}_i + \mathbf{Z}_i^T \qquad i \in \lbrace 1 \ldots x \rbrace$$ where $\mathbf{Z}_i$ is an all zero matrix except vector $x$ in its $i^{th}$ column.

Hessian

The derivative of $\frac{\partial \mathbf{Z}_i}{\partial x_j}$ is an all zero matrix except $1$ at its $(j,i)$ position. By symmetry, the derivative of $\frac{\partial \mathbf{Z}_i^T}{\partial x_j}$ is an all zero matrix except $1$ at its $(i,j)$ position. This means that $$\frac{\partial \mathbf{xx^T}}{\partial x_i \partial x_j} = \mathbf{K}_{i,j} \qquad (i,j) \in \lbrace 1 \ldots x \rbrace$$ where $\mathbf{K}_{i,j}$ is an $n \times n$ matrix which is all-zero except at positions $(i,j)$ and $(j,i)$. Note that if $i = j$, we get a $2$ in the $i^{th}$ (or $j^{th}$) element.

Answer 2

This is standard Matrix differentiation

The gradient is $2x$ and the Hessian is $2 I$.

By the way I'm assuming you meant $x^T x$ since $x x^T$ is a matrix and don't have a gradient or Hessian.

Answer 3

It's a cinch to calculate these things in index notation. $$\eqalign{ F_{ij} &= x_ix_j \cr G_{ijk} = \frac{\partial F_{ij}}{\partial x_k} &= x_i\delta_{jk} + x_j\delta_{ik} \cr H_{ijkl} = \frac{\partial^2F_{ij}}{\partial x_k\partial x_l} &= \delta_{il}\delta_{jk} + \delta_{ik}\delta_{jl} \cr }$$