I want to take a derivative with respect to $A$ and $B$ but I have a problem. The function is
$$f=\left\|\left(B^TA\right)\circ C\right\|_{F}^2$$ Where $\circ$ is Hadamrad product and $B^TA$ is a normal matrix multiplication. $B^TA$ and $C$ both are R by R matrix. Thank you.
On
To streamline the calculation, define the matrix $$M=C\circ(B^TA)$$ Write the function in terms of the inner/Frobenius product and this new matrix $$f = M:M$$ Then find an expression for the differential $$\eqalign{ df&= 2M:dM \cr &= 2M:C\circ d(B^TA) \cr &= 2(C\circ M):(dB^TA+B^TdA) \cr &= 2(C\circ M)A^T:dB^T + 2B(C\circ M):dA \cr &= 2A(C\circ M)^T:dB + 2B(C\circ M):dA \cr }$$ Therefore the gradients are $$\eqalign{ \frac{\partial f}{\partial A} &= 2B(C\circ M) \cr \frac{\partial f}{\partial B} &= 2A(C^T\circ M^T) \cr\cr }$$
$$B^t(A+ t a) \circ C = t B^t a \circ C.$$ Now, the Frobenius norm squared is just the inner product of the matrix with itself, so we get that the derivative is just $$2 (B^t a) \circ C.$$ All this because all the operations involved (except the final norm squared) are linear.