hadamard product differentiation

Question

How to differentiate the following formula with respect to $\bf Y$

$\bf b^T (Y \odot Y)c$

in which the $\bf b$ and $\bf c$ are vectors. $\bf Y$ is a non-square matrix. and $\odot$ denotes hadamard product.

Answer 1

Write the function in terms of the Frobenius Inner Product, then find its differential and gradient $$\eqalign{ f &= bc^T:Y\circ Y \cr\cr df &= bc^T: d\,(Y\circ Y) \cr &= bc^T: (2\,Y\circ dY) \cr &= (2\,Y\circ bc^T): dY \cr\cr \frac{\partial f}{\partial Y} &= 2\,Y\circ bc^T \cr\cr }$$ Now that you have the gradient, you want to press on and find the hessian. The problem is that the gradient is a matrix, therefore the hessian will be a 4th order tensor!

One way to proceed is vectorization $$\eqalign{ G &= (2bc^T)\circ Y \cr g &= {\rm vec}(G) \cr &= {\rm vec}(2bc^T)\circ y \cr &= {\rm Diag}({\rm vec}(2bc^T))\,y \cr\cr dg &= {\rm Diag}({\rm vec}(2bc^T))\,dy \cr\cr \frac{\partial g}{\partial y} &= {\rm Diag}({\rm vec}(2bc^T)) \cr\cr }$$ Another way to proceed is via Einstein-index notation. To make things easier we'll use a special 3rd-order tensor whose components $T_{ijk}=1$ when $i\!=\!j\!=\!k$ but are zero otherwise. Note that this tensor is symmetric wrt all of its indices. $$\eqalign{ G_{jn} &= 2 b_i c_m\,\, T_{ijk} T_{mnp}\,\, Y_{kp} \cr\cr \frac{\partial G_{jn}}{\partial Y_{rs}} &= 2 b_i c_m\,\, T_{ijk} T_{mnp}\,\, \delta_{kr} \delta_{ps} \cr &= 2 b_i c_m\,\, T_{ijr} T_{mns} \cr }$$