Determining the derivative of a multivariable function

Question

I am asked to determine the derivative of the function \begin{align*} f(\textbf{x}) = \left\|\textbf{A}\textbf{x}-\textbf{y}\right\|_{2} ^{2}+\alpha \textbf{x}^{\mathsf{T}}\textbf{M}\textbf{x} \end{align*} with $\textbf{A} \in \mathbb{R}^{m, n}, \ \textbf{x}, \textbf{y} \in \mathbb{R}^{n} , \ \textbf{M}\in \mathbb{R}^{m, n} , \ \alpha\in \mathbb{R}^{} $. I know that we can differentiate termwise and for the first I got: \begin{align*} \frac{\partial }{\partial x_{i}} \left\|\textbf{A}\textbf{x}-\textbf{y}\right\|_{2} &= \frac{\partial }{\partial x_{i}} \sum_{k=1}^{m} (\textbf{A}\textbf{x}-\textbf{y})_{k}^{2} =\frac{\partial }{\partial x_{i}} \sum_{k=1}^{n} \left(\sum_{j=1}^{n} a_{k, j}x_{j} - y_{k}\right)^{2} \\ &= \frac{\partial }{\partial x_{i}} \sum_{k=1}^{n} (a_{k, i}\cdot x_{i}-y_{k})^{2} = 2\sum_{k=1}^{n} a_{k, i}^{2}\cdot x_{i} - 2 \sum_{k=1}^{n} a_{k, i}\cdot x_{i}\cdot y_{k} .\end{align*}

Is my calculation above correct? Is there an easier way to determine the derivative of this expression? Thanks in advance!

Answer 1

Yes there is a simpler approach - you can use the derivative rules for vectors/matrices. For example some properties that would be useful here are $\nabla x^T y = y$. $\nabla x^T M x = 2 M x$

Further the norm term can be written as $(Ax-y)^T (Ax-y)$ and then you can distribute to apply these rules.

You can do what you are suggesting but its way more tedius

Answer 2

The differential writes \begin{eqnarray} df &=& 2(\mathbf{Ax-y}): \mathbf{A} d\mathbf{x}+ 2\alpha \mathbf{x}:\mathbf{M} d\mathbf{x} \\ &=& 2 \left[ \mathbf{A}^T(\mathbf{Ax-y})+ \alpha \mathbf{M}^T\mathbf{x} \right]: d\mathbf{x} \end{eqnarray} with the colon notation $\mathbf{a}:\mathbf{b}= \mathbf{a}^T \mathbf{b}$.

Thus the derivative can be expressed in matrix form as $$ \frac{\partial f}{\partial \mathbf{x}} = 2 \left[ \mathbf{A}^T(\mathbf{Ax-y})+ \alpha \mathbf{M}^T\mathbf{x} \right] $$