Derivative of L1 norm of Hadamard product

297 Views Asked by At

I am trying to find the derivative of $f(B)=\lambda\Vert W \bigodot B \Vert_1 + \frac{\rho}{2}\Vert A-B \Vert_F^2 + tr(\Delta^T(A-B))$ with respect to B.

where B is (n×n)matrix, W is (n×n)constant matrix, A is (n×n)constant matrix. $\lambda$ and $\rho$ are scalars. $tr$ is the trace of the matrix. $W \bigodot B$ is the Hadamard product of W and B.

I am troubled in finding the derivative involving Hadamard product and L-1 norm. Therefore, I first replaced $W \bigodot B$ with T. $$T=W \bigodot B$$

$$B=W^{-1} \bigodot T$$ where $W^{-1}$ is the element-wise inverse. $W \bigodot W^{-1}=I$.

$$f(T)=\lambda\Vert T \Vert_1 + \frac{\rho}{2}\Vert A-W^{-1} \bigodot T \Vert_F^2 + tr(\Delta^T(A-W^{-1} \bigodot T))$$

I do not konw what to do next. Thank you in advance for any help you can provide.

1

There are 1 best solutions below

1
On

Let's use a colon to denote the trace/Frobenius product, i.e. $\;A:B = {\rm Tr}(A^TB)$
The cyclic property of the trace allows such products to be rearranged in many different ways, e.g. $$\eqalign{ A:B &= A^T:B^T &= B:A \\ A:BC &= AC^T:B &= B^TA:C \\ }$$ Together, the Frobenius and Hadamard products form a scalar/triple product, whose terms commute. $$A:(B\odot C) = (A\odot B):C$$ Using these products and the element-wise sign function, we can calculate the differential and subgradient of the troublesome term. $$\eqalign{ \phi &= \lambda \|T\|_1 \\ &= \lambda\operatorname{sign}(T):T \\ d\phi &= \lambda\operatorname{sign}(T):dT \\ &= \lambda\operatorname{sign}(T):W\odot dB \\ &= \lambda W\odot\operatorname{sign}(T):dB \\ &= \lambda W\odot\operatorname{sign}(W)\odot\operatorname{sign}(B):dB \\ \frac{\partial\phi}{\partial B} &= \lambda W\odot\operatorname{sign}(W)\odot\operatorname{sign}(B) \\ }$$ Introducing an element-wise absolute value function, we can write this as $$\eqalign{ \frac{\partial\phi}{\partial B} &= \lambda\operatorname{abs}(W)\odot\operatorname{sign}(B) \\ }$$