So if I define $y(U)=|Tr(U^*V)|^2$
If I do the Gateaux derivative:
$y_U[\tilde{U}] =\frac{d}{d\epsilon}|Tr((U^*+\epsilon\tilde{U})V)|^2 $,
Here is where I get confused because of how things are nested - the trace has a real and imaginary part and I'm not sure how to handle this
My intuition says that the derivative is like $y_U[\tilde{U}] = 2Tr(V^*)$ with the variation somewhere in here, but I can't figure out how to get that if it should be the case. Some tips would be much appreciated
$y(U) = \overline{\operatorname{tr} (U^*V)} \operatorname{tr} (U^*V)$.
\begin{eqnarray} y(U+tH) -y(U) &=& \overline{\operatorname{tr} ((U+tH)^*V)} \operatorname{tr} ((U+tH)^*V) - \overline{\operatorname{tr} (U^*V)} \operatorname{tr} (U^*V) \\ &=& (\overline{\operatorname{tr} (U^*V)} + t\overline{\operatorname{tr}(H^*V)} ) ( \operatorname{tr} (U^*V) + t\operatorname{tr} (H^*V) ) - \overline{\operatorname{tr} (U^*V)} \operatorname{tr} (U^*V) \\ &=& 2 t \operatorname{re} ( \overline{\operatorname{tr} (U^*V)} \operatorname{tr} (H^*V) ) + O(t^2) \end{eqnarray}
Hence $d y(U,H) = 2 \operatorname{re} ( \overline{\operatorname{tr} (U^*V)} \operatorname{tr} (H^*V) ) $.
Note that there are many different notations for the Gateaux derivative.
Note, if $H$ is real, with $\operatorname{tr} (U^*V) = a+ib$ and $V= V_r+i V_i$ then $d y(U,H) = 2 \operatorname{re} ( (a-ib) ( \operatorname{tr} (V_i^TH) -i \operatorname{tr} (V_r^TH)) = 2 \operatorname{tr}((aV_r-bV_i)^T H)$, and similarly, $d y(U,iH) = 2 \operatorname{re} ( (a-ib) ( \operatorname{tr} (V_i^TH) -i \operatorname{tr} (V_r^TH)) = 2 \operatorname{tr}((bV_r+aV_i)^T H)$