Let $A\in\mathbb{R}^{m\times n}$ and $b\in\mathbb{R}^m$, $m\ge n$. The normal equation for minimizing $\| Ax-b\|_2^2$ with respect to $x$ is $A^TAx=A^Tb$. Then derive the normal equation for minimizing $\| Ax-b\|_C^2$, where C is a symmetric positive definite matrix and $\| x\|_C^2=x^TCx$.
I don't know how to start with; can someone give me some hint about this? Thank you.
The vectors in $\mathbb{R}^n$ and $\mathbb{R}^m$ are column vectors. If you consider $f(y)=\|y\|_2^2$ in $\mathbb{R}^m$ then the derivative is $$ f'(y)=2(y_1,\dots,y_m) = 2y^\top. $$ For $g:\mathbb{R}^n \to \mathbb{R}^m$, $g(x)=Ax-b$ we have $g'(x)=A$. Now the chain rule gives $$ (f\circ g)'(x)=2(Ax-b)^\top A. $$ So, the critical points are those $x$ with $(Ax-b)^\top A=0$. Transposing leads to $A^\top(Ax-b)=0$, that is $A^\top Ax=A^\top b$.
For $C$ positive definite try to find the derivative of $x \mapsto x^\top C x$.