Consider the quadratic cost function
$$ J(U) = \tfrac{1}{2}U^HAU+U^HBa-2U^HC b, $$ where $U\in\ \mathbb{C}^{81}$ is the vector of decision variables and the other matrices ($a$, $b$ are vectors) are constant and of appropriate dimension. This is basically a quadratic cost function. I want to optimize this in MATLAB and provide its gradient $\nabla J_U$. So, if i am not mistaken $$ \nabla J_U=AU+a^HB^H-2b^HC^H, $$ note that $A$ is an hermitian matrix. The main issue is that $U$ is an intermediate variable, namely $$ U = \text{vec}\ \overline{q}q^\top, $$ where $\overline{(\cdot)}$ denotes the conjugate, $(\cdot)^H$ the conjugate transpose and $(\cdot)^\top$ the non-conjugate transpose. Furthermore, $U\in\mathbb{C}^{9}$. The gradient $\nabla U_{q_1, \dots, q_9}$ is rather cumbersome, but calculable. My question is, how to connect these results with each other? I am not very good at vector calculus but it is obvious that i require the chain-rule here. But i am not certain how to apply it here.
EDIT Let me give you the origin of the equation. I start with $$ \begin{bmatrix}q^HE_1q\\ \vdots \\ q^hE_nq\end{bmatrix}=\begin{bmatrix}\text{vec}\ (E_1)^\top\\ \vdots \\ \text{vec}\ (E_n)^\top\ \end{bmatrix}\text{vec}\ (\overline{q}q^\top)=\Gamma u $$
$E_i$ is hermitian, hence the left term will always be real. In my scenario, you can never arbitrarily multiply $u$ with a complex scalar, because $u$ follows from $q$.
For convenience, define the vectors $$\eqalign{ b &= Ba,\qquad c=-2Cb \\ y &= \tfrac{1}{2}Au,\,\quad v = y+b+c \\ }$$ and the matrix $$\eqalign{ V &= {\rm Reshape}(v,9,9) \quad\implies\quad v = {\rm vec}(V) \\ Y &= {\rm Reshape}(y,9,9) \quad\implies\quad y = {\rm vec}(Y) \\ }$$ Let's use a colon to denote the trace/Frobenius product, i.e. $$\eqalign{ &A:B = {\rm Tr}(A^TB) = {\rm Tr}(B^TA)=B:A \\ &A:BC = B^TA:C = AC^T:B = etc \\ }$$ Then your cost function can be written as $$\eqalign{ J &= y^Hu + b^Hu + c^Hu \\ &= (y+b+c)^*:u \\ &= v^*:u \\ }$$ Calculate the differential of $J$ $$\eqalign{ dJ &= v^*:du + u:dv^* \\ &= v^*:du + u:\left(\tfrac{1}{2}A^*du^*\right) \\ &= v^*:du + y:du^* \\ }$$ Next, utilize the relationship with $q$
$$\eqalign{ u &= {\rm vec}(q^*q^T) \\ du &= {\rm vec}(q^*dq^T+dq^*q^T) \\ du^* &= {\rm vec}(q\,dq^H+dq\,q^H) \\ }$$ Substitute this into the differential expression, then calculate the gradient $$\eqalign{ dJ &= v^*:du + y:du^* \\ &= {\rm vec}(V^*):{\rm vec}(q^*dq^T+dq^*q^T) + {\rm vec}(Y):{\rm vec}(q\,dq^H+dq\,q^H) \\ &= V^*:\left(q^*dq^T+dq^*q^T\right) + Y:\left(q\,dq^H+dq\,q^H\right) \\ &= \left(V^H:dq\,q^H + V^*:dq^*q^T\right) + \left(Y^T:dq^*q^T + Y:dq\,q^H\right) \\ &= \left(V^Hq^*:dq + V^*q:dq^*\right) + \left(Y^Tq:dq^* + Yq^*:dq\right)\\ &= \left(V^Hq^*+Yq^*\right):dq + \left(V^*q+Y^Tq\right):dq^* \\ \frac{\partial J}{\partial q} &= V^Hq^* + Yq^*, \qquad \frac{\partial J}{\partial q^*}= V^*q + Y^Tq \\ }$$ Since $J$ is real, these results implies that $(V,Y)$ are hermitian.
Some comments asked about how the trace/Frobenius product behaves with vectors. $$\eqalign{ {\rm Tr}(y^TAx) &= {\rm Tr}((Ax)^Ty) &= {\rm Tr}((yx^T)^TA) &= {\rm Tr}(A(yx^T)^T) &= etc \\ y:Ax &= Ax:y &= yx^T:A &= A:yx^T &= etc \\\\ }$$
Update
The Frobenius Inner Product described on Wikipedia is defined as $$\langle A,B\rangle = {\rm Tr}(A^HB)$$ and is different from the product used in this answer which is $$A:B={\rm Tr}(A^TB)$$ The colon product has the following extremely useful properties $$\eqalign{ A:B &= B:A \qquad&\big({\rm commutes\,with\,itself}\big) \\ A:B\odot C &= A\odot B:C \qquad&\big({\rm commutes\,with\,Hadamard}\big) \\ }$$ which hold even when the matrices involved are complex. The Wikipedia product satisfies neither property.I find the fact that $\langle A,B\rangle \ne \langle B,A\rangle$ and trying to remember which matrix gets conjugated to be a perpetual source of errors. The only nice property of the Wikipedia product is $$\|A\|_F^2 = \langle A,A\rangle$$ but this only fractionally more succinct than $$\|A\|_F^2 = A^*:A$$