I am seeking to compute the following:
$\nabla ( \mathbf{1}^T f(X) )$
Where:
$\mathbf{1}^T = (1 , \ldots , 1)$ an $1 \times n$ vector.
$f(X)= (A*X) \circ (B*X)$
$A$ and $B$: $n \times n$ matrices
$X$: $n \times 1$ vector
$\circ$ the Hadamard or element wise product
$*$ matrix product.
I read the following rule $d(X \circ Y) = d(X) \circ Y + d(Y) \circ X$
But in the above I have a mixture of matrix and element wise product (dimensions are consistent at the end $\mathbf{1}^T f(X) $ is of size $1 \times 1$)
Let's denote the elementwise/Hadamard and inner/Frobenius products respectively as $$\eqalign{ A &= B\circ C \cr \alpha &= B:C = {\rm tr}(B^TC) \cr }$$ Recall that these products are commutative, and mutually commutative $$\eqalign{ A\circ B &= B\circ A \cr A:B &= B:A \cr A:B\circ C &= A\circ B:C \cr }$$ and that the matrix of all ones is the identity element for the Hadamard product. Note that the matrices $(A,B,C)$ must have the same shape for these products to make sense.
Your scalar function can be written as $$\eqalign{ y &= 1:f \cr &= 1:(Ax)\circ(Bx) \cr &= Ax:Bx \cr }$$ Whosee differential and gradient are $$\eqalign{ dy &= A\,dx:Bx + Ax:B\,dx \cr &= Bx:A\,dx + Ax:B\,dx \cr &= (A^TB + B^TA)\,x:dx \cr \cr \frac{\partial y}{\partial x} &= (A^TB + B^TA)\,x \cr\cr }$$ (Note that I've used juxtaposition rather than $*$ for the ordinary matrix product.)