I want to find the gradient of $f : \mathbb{C}^{N} \rightarrow \mathbb{R}$, where
$$ f(\mathbf{x}) = \frac 1 2 \Vert \mathbf{Ax} - \mathbf{b} \Vert_2^2, $$
and $\mathbf{A}\in\mathbb{C}^{M \times N}$, $\mathbf{x}\in\mathbb{C}^{N}$, $\mathbf{b}\in\mathbb{C}^{M}$.
For real matrices, it's straightforward, but I am getting stuck with the complex case:
$$ f(\mathbf{x}) = \frac 1 2 ( \mathbf{Ax} - \mathbf{b} )^\mathrm{H} ( \mathbf{Ax} - \mathbf{b} ) = \frac 1 2 ( \mathbf{x}^\mathrm{H} \mathbf{A}^\mathrm{H} \mathbf{Ax} - \mathbf{x}^\mathrm{H} \mathbf{A}^\mathrm{H} \mathbf{b} - \mathbf{b}^\mathrm{H} \mathbf{Ax} + \mathbf{b}^\mathrm{H} \mathbf{b} ). $$
For the first and thirds terms, the gradient is $2 \mathbf{A}^\mathrm{H} \mathbf{Ax}$ and $\mathbf{b}^\mathrm{H} \mathbf{A}$, respectively. For the fourth term it is $\mathbf{0}$. However, it's the gradient of the second term that has me puzzled and I am wondering if there is something fundamental that I have forgotten. How do I account for the conjugation of $\mathbf{x}$ in the gradient? Is that even possible?
Any help is greatly appreciated.
For a matrix $M$, let me denote its complex conjugate by $M^*$, its transpose by $M^T$ and its hermitian conjugate by $M^H=M^{*T}$.
Write the function in terms of the Frobenius (:) Inner Product and take its differential $$\eqalign{ f &= \frac{1}{2}(Ax-b)^*:(Ax-b) \cr\cr df &= \frac{1}{2}\Big(A^*\,dx^*:(Ax-b) + (Ax-b)^*:A\,dx\Big) \cr &= \frac{1}{2}\Big(dx^*:A^H(Ax-b) + A^T(Ax-b)^*:dx\Big) \cr\cr }$$ Now the tricky part is to treat $dx^*$ and $dx$ as independent variables.
To find the gradient wrt $x$ you hold $x^*$ constant, i.e. set $\,dx^*=0$, to obtain $$\eqalign{ \frac{\partial f}{\partial x} &= \frac{1}{2}A^T(Ax-b)^* \cr\cr }$$ While holding $x$ constant yields $$\eqalign{ \frac{\partial f}{\partial x^*} &= \frac{1}{2}A^H(Ax-b) \cr\cr }$$ Notice that the two expressions are complex conjugates of one another, so you only need calculate one of them.