what is the gradient of $||Ax -b||^2$?

864 Views Asked by At

I was trying to figure out the gradient descent approach of solving Ax-b for x. I have previously seen the closed form solution for x by minimizing $||Ax - b||^2$;

That gives the equation $x = (A^T A)^{-1} A^T b$

I was trying to derive this myself; however I see some conflicting answers. check out the answer given here. The accepted answer, and the one below it, give different gradients.

  1. gradient = $2A^T (Ax - b)$
  2. gradient = $2(Ax - b)^T A$

The 2nd equation is transpose of the 1st. Is the second equation wrong? What is the gradient of the given cost function? is it the first equation?

However, when I minimize it by equating it to 0, I get the same closed form equation for x

i.e, $x = (A^T A)^{-1} A^T b$

I am not very familiar with vector calculus, and I might be asking a novice question. Any incite is helpful, thanks :)