The following identity
$$\langle \frac{\partial f}{\partial x}\rangle \sigma^2 = \langle (x-\mu) f(x)\rangle,$$
where $x$ is a Gaussian random variable with $x \sim N(\mu, \sigma^2)$, and $f(x)$ is any smooth function of the random variable, was mentioned and used in the following paper "The Variational Gaussian Approximation Revisited." by Opper and Archambeau. (page 5 above eq. 13)
The result is remarkable as we do not even need the gradient of log likelihood function (or loss function) w.r.t the parameter when approximating the gradient of variational parameter by Monte Carlo method.
I have been trying to prove it but have no clue. For some trivial choices of $f$, say $f(x) = x$, the identity is obviously correct. Simulations also show the identity perform well for a wide range choices of $f$.
This is known as the Gaussian integration by parts formula. Let $X \sim \mathrm{N}(\mu, \sigma^2)$. Then $X$ has density $\varphi(x) = \frac{1}{ \sqrt{2 \pi} \sigma} \exp\left( -\frac{(x - \mu)^2}{2 \sigma^2} \right)$. Check that $(x - \mu) \varphi(x) = -\sigma^2 \varphi'(x)$. Therefore, for any sufficiently smooth function, we can do integration by parts to obtain $$ \mathbb{E} (X - \mu) f(X) = \int (x - \mu) \, f(x) \, \varphi(x) \;\mathrm{d} x = \left.-\sigma^2 f(x) \, \varphi(x) \right|^\infty_{-\infty} + \sigma^2 \int f'(x) \, \phi(x) \; \mathrm{d} x = \sigma^2 \mathbb{E} f'(X) . $$