How do you compute L, the function whose partial derivative with respect to W is as below?
Where B and W are matrices, and e and h are vectors. f is a pointwise nonlinear, but differentiable function. I am pretty new to matrix calculus, but I couldn't find anything in The Matrix Cookbook that I could directly use.
For convenience, define the variables
$$\eqalign{ x &= Wh \cr y &= Be \cr Y &= {\rm Diag}(y) \cr g=g(x) &= f^\prime(x) \cr \cr }$$ Now reformulate the differential of $L$ $$\eqalign{ dL &= \frac{\partial L}{\partial W}:dW \cr &= Ygh^T:dW \cr &= Yg:dWh \cr &= Yg:dx \cr &= y\odot g:dx \cr &= y:g\odot dx \cr &= y:df \cr }$$ Now we can integrate $$\eqalign{ L &= \int dL \cr &= y:\int df \cr &= y:f \cr &= y^Tf \cr\cr }$$ We can pull $y$ out of the integral because it is independent of $(W,h,x)$.
In some of the steps, we made use of the fact that the elementwise/Hadamard product (denoted by $\odot$) and the inner/Frobenius product (denoted by $:$) commute with themselves and each other.
For complete generality, we should include a constant of integration $$L = y^T(f+c)$$