Derivative of $\|Ax+b\|_2^2c$ w.r.t. $x$

Question

What is the derivative of this expression with respect to x? Here, $c$ is a column vector.

$||Ax+b||^2_2c$

I think it has to be of the form $2A^T(Ax+b)c^T$ or $2c(Ax+b)^TA$ but I don't know how to arrive at this answer.

Edit: The original function I was working on is of the form $$f(x)=\frac{||Ax+b||^2_2}{c^Tx+d}$$ and I was trying to derive its Hessian.

Answer 1

Background info: There's a rule for taking derivatives in multivariable calculus which states that if $g:\mathbb R^n \to \mathbb R^m$ is differentiable and $C$ is a $k \times m$ matrix, then the derivative of the function $f(x) = C g(x)$ is $f'(x) = C g'(x)$. We can apply this rule in this problem.

---

In this case, your function can be written as $$ f(x) = c g(x) $$ where $g(x) = \| Ax + b \|_2^2$. The derivative of $f$ is \begin{align*} f'(x) &= c g'(x) \\ &= 2c(Ax + b)^T A . \end{align*}

---

Computing the derivative of the function $g(x) = \| Ax + b \|_2^2$ is a question that comes up frequently on this site. I think the most elegant way to do it is using the chain rule. Note that $g(x) = h(u(x))$, where $u(x) = Ax + b$ and $h(v) = \| v \|_2^2$. The derivatives of $h$ and $u$ are $u'(x) = A$ and $h'(v) = 2v^T$. So by the chain rule $$ g'(x) = h'(u(x)) u'(x) = 2(Ax + b)^T A. $$

Answer 2

Okay, as far as I understand, A is a matrix, b is a vector of appropriate size, and c is a vector of arbitrary size. The vector $x$ is the only variable in this equation.

In this case, we can simply compute the derivative of $\frac{\partial}{\partial x}\|Ax+b\|^{2}$ and then multiply the result by the column-vector $c$.

By the rules of matrix calculus, $$ \frac{\partial}{\partial x}\|Ax+b\|^{2} = \frac{\partial}{\partial x} (x^\top A^\top Ax +2 b^\top Ax+\|b\|^{2})=2A^\top A x+2A^\top b. $$

We need to be careful that the dimensions work out well. So let's make our derivative vector a row vector by taking the transpose. Hence, $$ \bigg(\frac{\partial}{\partial x} \|Ax+b\|^{2} \bigg)^\top c = 2(A^\top A x+A^\top b)^\top c. $$

Perhaps you're unfamiliar with matrix calculus. Here's the wikipedia article covering it: https://en.wikipedia.org/wiki/Matrix_calculus

It's just a matter of getting used to the notation. The only tricky result to show is that $\frac{\partial }{\partial x} x^\top Mx = 2Mx$ for any matrix $M$ and vector $x$ of appropriate dimensions.