I have a function as following
$$F=\int |A^TG(x)-B^TJ(x)|^2 H(x)\,dx+\lambda_1 A^2+\lambda_2 B^2$$
where $A^T$ is transpose of vector $A$. $A$ is a column vector such as $A= \begin{bmatrix} a_1 \\ a_2 \\ ... \\ a_n \\ \end{bmatrix} $; $B= \begin{bmatrix} b_1 \\ b_2 \\ ... \\ b_n \\ \end{bmatrix} $; $A^TG(x)=\sum_{i=1}^{i=n} a_i G_i $
$G=[G_1...G_i..G_n],J=[J_1...J_i..J_n]$, in which $G_i$ is matrix $n\times n$ and $H(x)$ is Heaviside smoothing function. Note that given $G,J,H$
However, How to find the derivative of these above function?
$$\frac {\partial F}{\partial A}=?$$ $$\frac {\partial F}{\partial B}=?$$ This is my trying
$$\frac {\partial F}{\partial A}=2\int G(x)(A^TG(x)-B^TJ(x)) H(x)\,dx+2\lambda_1 A=0$$
Since {$G,J$} are third-order tensors, I prefer to write explicit dot products in expressions involving them.
Let $M_a = G\cdot A$ and $M_b = J\cdot B$ and $q = \|M_a-M_b\|^2_F$.
Then the function and its partial derivative are $$\eqalign{ F &= \lambda_1\|A\|^2 + \lambda_2\|B\|^2 + \int q(x)\,H(x)\,dx \cr\cr \frac {\partial F}{\partial A} &= 2\,\lambda_1 A + \frac{\partial}{\partial A}\int q(x)\,H(x)\,dx \cr &= 2\,\lambda_1 A + \int \frac {\partial q} {\partial A}\,H(x)\,dx \cr }$$ Now we need the differential / derivative of $q$ $$\eqalign{ q &= (M_a-M_b):(M_a-M_b) \cr\cr dq &= 2\,(M_a-M_b):dM_a \cr &= 2\,(M_a-M_b):G\cdot dA \cr\cr \frac {\partial q} {\partial A} &= 2\,(M_a-M_b):G \cr &= 2\,(G\cdot A-J\cdot B):G \cr }$$ The final result is $$\eqalign{ \frac {\partial F}{\partial A} &= 2\,\lambda_1 A + 2\,\int (G\cdot A-J\cdot B):G\,H(x)\,dx \cr }$$
This result differs from yours because the presence of the double-dot (aka Frobenius) product.
A similar result holds for the $B$ vector.