I was trying to figure out the gradient descent approach of solving Ax-b for x. I have previously seen the closed form solution for x by minimizing $||Ax - b||^2$;
That gives the equation $x = (A^T A)^{-1} A^T b$
I was trying to derive this myself; however I see some conflicting answers. check out the answer given here. The accepted answer, and the one below it, give different gradients.
- gradient = $2A^T (Ax - b)$
- gradient = $2(Ax - b)^T A$
The 2nd equation is transpose of the 1st. Is the second equation wrong? What is the gradient of the given cost function? is it the first equation?
However, when I minimize it by equating it to 0, I get the same closed form equation for x
i.e, $x = (A^T A)^{-1} A^T b$
I am not very familiar with vector calculus, and I might be asking a novice question. Any incite is helpful, thanks :)