Differentiating with Hadamard: $Ax \odot b$

Question

I'm trying to differentiate $\mathbf{A}\mathbf{x}\odot \mathbf{b}$, with respect to $\mathbf{x}$, where $\odot$ is the Hadamard/entrywise product.

I tried making a simple example where $\mathbf{A}\in \mathbf{R}^{2\times2}$, and multiplying out: $$\left[ {\begin{array}{cc} a_{11} & a_{12} \\ a_{21} & a_{22} \end{array}} \right] \left[ {\begin{array}{c} x_1 \\ x_2 \end{array}} \right] \odot \left[ {\begin{array}{c} b_1 \\ b_2 \end{array}} \right] = \left[ {\begin{array}{cc} b_1(a_{11}x_1+a_{12}x_2) \\ b_2(a_{21}x_1+a_{22}x_2) \end{array}} \right]$$ Then, $\frac{d}{d\mathbf{x}}$ is $$\left[ {\begin{array}{cc} b_1a_{11} & b_1a_{12} \\ b_2a_{21} & b_2a_{22} \end{array}} \right]$$

In matrix notation that looks like $\mathbf{A}$ entry wise multiplied by some "double-column" $\mathbf{b}$, but I can't see what this would be.

Is this correct? And how would you write this in matrix notation?

Answer 1

The Hadamard product between vectors can be written in a number of ways $$\eqalign{ v\circ b &= b\circ v &= Vb &= Bv \cr }$$ where $\,\,B={\rm Diag}(b)$ and $\,V={\rm Diag}(v)$.

Which allows you to write the function in a form which is easier to work with $$\eqalign{ y &= b\circ(Ax) \cr &= B\,(Ax) \cr &= BA\,x \cr\cr dy &= BA\,dx \cr\cr \frac{\partial y}{\partial x} &= BA \cr &= {\rm Diag}(b)A \cr\cr }$$

Answer 2

In index notation, $$ (\mathbf{A}\mathbf{x}\odot \mathbf{b})_i=(\mathbf{A}\mathbf{x})_i b_i=\sum_{j}A_{ij}x_jb_i, $$ so $$ \frac{\partial}{\partial x_{j}}(\mathbf{A}\mathbf{x}\odot \mathbf{b})_i=b_i A_{ij}; $$ i.e., the derivative matrix is ${\mathbf{A}}$, with each row multiplied by the corresponding element of $\mathbf{b}$. As pointed out in the other answer, this is equivalent to $$\nabla_x (\mathbf{b} \odot \mathbf{A}\mathbf{x}) = \text{diag}(\mathbf{b}) \mathbf{A}.$$