Gradient of $f(x) = (Ax - b)^\mathsf{T} (Ax - b)$

Question

I want to find the gradient of $f(x) = (Ax - b)^\mathsf{T} (Ax - b)$, from product rule, I got:

$$ \mathrm{d}f(x) = (A \mathrm{d}x)^\mathsf{T}(Ax-b)+(Ax-b)^\mathsf{T}(A \mathrm{d}x) $$ now I know I need to reformat the RHS to look like (something)$^\mathsf{T} \mathrm{d}x$, so that the something is the gradient I need to find. But I have trouble figuring how to get it to this form.

Answer 1

The linked question shows you how to take the derivative in coordinates, but I much prefer your approach using the differential. The last step you need is the observation that a scalar is always equal to its own transpose; you can use this to reorder the (scalar) first term so that the $dx$ goes to the right.

Answer 2

Check out the attachment that I attached to the answer. Answer