Derivative of Hadamard product

Question

What is the derivative of Hadamard product of two matrices with respect to one of them? I.e. what is $D(AB)$ with respect to $A$?

Answer 1

For the derivative with respect to one component of $A$: $$\frac{\partial}{\partial a_{ij}} (AB)_{kl} = \frac{\partial}{\partial a_{ij}} a_{kl}b_{kl} = b_{kl}\delta_{ik}\delta_{jl}.$$

You'll then have to pick some representation for writing down the full derivative; it is a rank four tensor.

Answer 2

Consider the 6th-order tensor ${\mathbb M}$, whose components ${\mathbb M}_{ijklmn}$ are unity if $\,(i=k=m)$ and $(j=l=n),\,$ but zero otherwise.

This tensor can be used, along with the Frobenius Product (:), to rewrite the Hadamard Product ($\circ$) between two matrices as
$$\eqalign{ B\circ A &= B:{\mathbb M}:A \cr\cr }$$ This makes the finding the gradient (a 4th-order tensor) especially simple $$\eqalign{ F &= A\circ B = B\circ A = B:{\mathbb M}:A \cr\cr dF &= B:{\mathbb M}:dA \cr\cr \frac{\partial F}{\partial A} &= B:{\mathbb M} \cr }$$

Answer 3

Let $f:A\in M_n\rightarrow A\circ B\in M_n$. $f$ is linear ! Then $Df_A:H\in M_n \rightarrow H\circ B$.

EDIT. @ bordeo , the derivative is a linear application; $H$ is any matrix of $M_n$. In particular, the Taylor formula can be written: $f(A+H)=f(A)+Df_A(H)+\cdots$, that is $(A+H)\circ B=A\circ B+H\circ B+\text{"nothing"}$; "nothing" because $f$ is linear and the terms of degree $\geq 2$ in the Taylor formula are $0$.