How to compute derivative with Hadamard product?

Question

Let $\mathbf{x}$, $\mathbf{y}$ and $\mathbf{z}$ are $n$-dimensional column vector, and

$$f = \mathbf{x}\circ \mathbf{y} \circ\mathbf{z}$$

Here $\circ$ is the element-wise Hadamard product. Then how to compute the gradient

$\frac{\partial f }{\partial\mathbf{x}}$, $\frac{\partial f }{\partial\mathbf{y}}$, and $\frac{\partial f }{\partial\mathbf{z}}$?

My solution is

$$\frac{\partial f }{\partial\mathbf{x}} = \mathbf{y} \circ\mathbf{z}, \frac{\partial f }{\partial\mathbf{y}} = \mathbf{x} \circ\mathbf{z},\frac{\partial f }{\partial\mathbf{z}} = \mathbf{x} \circ\mathbf{y}$$

Is it correct? thanks.

Answer 1

You can replace the Hadamard product of a vector by a matrix product with the diagonal matrix formed from that vector, e.g. let $$\eqalign{ X &= {\rm Diag}(x), \quad Y &= {\rm Diag}(y), \quad Z &= {\rm Diag}(z) \\ }$$ then the function can be written in a number of ways $$\eqalign{ f &= x\circ y\circ z \\&= YZx = ZXy = XYz \\ }$$ The differential of the function is $$\eqalign{ df &= YZ\,dx + ZX\,dy + XY\,dz \\ }$$ yielding the gradients $$\eqalign{ \frac{\partial f}{\partial x} &= YZ, \quad \frac{\partial f}{\partial y} &= ZX, \quad \frac{\partial f}{\partial z} &= XY \\ }$$ Since diagonal matrices commute, the order of multiplication can be reversed.

Answer 2

If $f\colon \Bbb R^n\times \Bbb R^n \times \Bbb R^n \to \Bbb R^n$ is given by $f({\bf x}, {\bf y}, {\bf z}) = {\bf x}\circ {\bf y} \circ {\bf z}$, then $f$ is trilinear. Thus its total derivative is the linear map ${\rm D}f({\bf x},{\bf y},{\bf z}): \Bbb R^n \times \Bbb R^n \times \Bbb R^n \to \Bbb R^n$ given by $${\rm D}f({\bf x},{\bf y}, {\bf z})({\bf h}_1,{\bf h}_2, {\bf h}_3) = {\bf h}_1 \circ {\bf y} \circ {\bf z} + {\bf x}\circ {\bf h}_2 \circ {\bf z} + {\bf x}\circ {\bf y} \circ {\bf h}_3. $$So we get that $$\frac{\partial f}{\partial {\bf x}}({\bf x},{\bf y},{\bf z}){\bf h} \equiv {\rm D}_{\bf x}f({\bf x}, {\bf y},{\bf z})({\bf h}) = {\bf h}\circ {\bf y} \circ {\bf z},$$ and similarly for the other partial derivatives. Since $\circ$ is commutative, you can write $$\frac{\partial f}{\partial {\bf x}}({\bf x},{\bf y},{\bf z}) = {\bf y}\circ {\bf z}\circ \_.$$

Answer 3

To take the derivative of the Hadamard product, follow equation 30 in the following paper

"Matrix differentiation with diagrammatic notation" https://arxiv.org/pdf/2207.04377.pdf