Derivative of Hadamard Product of two vectors

Question

How can I compute the following derivative? $$\frac{\partial(K u \circ T u)}{\partial u}$$ $K$ and $T$ are constant matrices, $u$ is an unknown vector. and $\circ$ is Hadamard product.

my solution: $$\frac{\partial(K_{ij} u_j \circ T_{mn} u_n)}{\partial u_p} = K_{ij}\frac{\partial(u_j)}{\partial u_p}\circ T_{mn} u_n+K_{ij} u_j \circ T_{mn} \frac{\partial u_n}{\partial u_p} = K_{ij}\delta_{jp} \circ T_{mn} u_n+K_{ij} u_j \circ T_{mn} \delta_{np}=K_{ip} (\sum_n T_{mn} u_n)+T_{mp} (\sum_j K_{ij} u_j).$$ therefore it can be written as follow $$\frac{\partial(K u \circ T u)}{\partial u} = K^T(Tu)+T^T(Ku).$$ where $\square^T$ is the transpose of the matrix.

Answer 1

Assuming $K,T\in\mathbb R^{n\times n}$ and $u\in\mathbb R^n\simeq\mathbb R^{n\times 1}$, we have (where $\square_{i,j}$ denotes the $(i,j)$-th entry of the matrix $\square$ and $\square^i=\square_i$ denotes the $i$-th entry of the vector $\square$), with the Einstein summation convention within $(\cdot)$,

$$(Ku\cdot_{\text{Hadamard}}Tu)_{i}=(K_{i,k}u^k)(T_{i,k}u^k),$$

so

$$\frac{\partial(Ku\cdot_{\text{Hadamard}}Tu)_{i}}{\partial u^j} = (K_{i,k}u^k) \partial_{u^j}[(T_{i,k}u^k)]+\partial_{u^j}[(K_{i,k}u^k)](T_{i,k}u^k)=(K_{i,k}u^k)T_{i,j}+(T_{i,k}u^k) K_{i,j}.$$

So far we agree.

I disagree with this resulting in the Jacobian of $Ku\cdot_{\text{Hadamard}}Tu$ being equal to $K^\top(Tu)+T^\top(Ku)$. For one, the Jacobian of $Ku\cdot_{\text{Hadamard}}Tu$ is an element of $\mathbb R^{n\times n}$ while $K^\top(Tu)+T^\top(Ku)\in\mathbb R^n$.

Correct is

$$\operatorname{Jac}Ku\cdot_{\text{Hadamard}}Tu = (\underbrace{Ku, Ku, \dots, Ku}_{n\text{ times}})\cdot_{\text{Hadamard}}T+(\underbrace{Tu, Tu, \dots, Tu}_{n\text{ times}})\cdot_{\text{Hadamard}}K.$$

Answer 2

The Hadamard product of two vectors is like a scalar product but without summing over the index. So in your "my solution" the index m should be i (but there is no sum over i). The last expression in that calculation is then correct.

You cannot rewrite this in your very last expression since there is no sum over i. If you take the directional derivative, i.e. the derivative in the direction of some vector $h$ then you may rewrite it avoiding the index notation: $$ \frac{\partial(K u \circ Tu)}{\partial u} . h = Kh \circ Tu + Ku \circ Th$$

Answer 3

$ \def\p{\partial} \def\L{\left}\def\R{\right}\def\LR#1{\L(#1\R)} \def\Diag#1{\operatorname{Diag}\LR{#1}} \def\qiq{\quad\implies\quad} \def\grad#1#2{\frac{\p #1}{\p #2}} $For typing convenience, define the vectors $$\eqalign{ a &= Ku &\qiq da=K\,du \\ b &= Tu &\qiq db=T\,du \\ }$$ Since a Hadamard product with a vector can be replaced by a standard product with a diagonal matrix, it will also be convenient to define the matrices $$A = \Diag a,\qquad B = \Diag b$$ Use the above notation to rewrite the Hadamard product and calculate its differential and gradient $$\eqalign{ p &= a\circ b \;= Ab \;= Ba \\ dp &= A\,db + B\,da = (AT + BK)\,du \\ \grad pu &= AT + BK \\ }$$