I would like to request a proof verification.
I am currently reading this paper, and I am trying to prove Eq. (3) from the paper.
The operator $\ast$ is the convolution operation and $\phi_n(x) = n\phi(nx)$ with $\phi(x) = \frac{1}{\sqrt{2\pi}}e^{-\frac{x^2}{2}}$. (Gaussian function)
I have tried this tedious calculation 3 times, and could not get the same answer as the paper. Can anyone check my calculations? I have written the full calculation process below.
I want to calculate the following function defined by a convolution. $$G(x, \alpha, n) = (\mathrm{LeakyReLU[\alpha]} \ast \phi_n)(x) = \int_{-\infty}^{\infty} \mathrm{LeakyReLU}[\alpha](x-y)\phi_n(y)\,d{y}.$$
Since
$$\mathrm{LeakyReLU}[\alpha](x-y) = \begin{cases} x - y & (y \leq x) \\ \alpha(x-y) & (y > x) \end{cases},$$
we split the integral into two parts.
$$ \begin{aligned} G(x, \alpha, n) &= \int_{-\infty}^{x} (x-y)\phi_n(y)\,d{y} + \int_{x}^{\infty} \alpha(x-y)\phi_n(y)\,d{y} \\ &= \int_{-\infty}^{x} (x-y)n\phi(ny)\,d{y} + \int_{x}^{\infty} \alpha(x-y)n\phi(ny)\,d{y} \\ &= \frac{n}{\sqrt{2\pi}} \int_{-\infty}^{x} (x-y) e^{-\frac{n^2y^2}{2}}\,dy + \frac{\alpha n}{\sqrt{2\pi}} \int_{x}^{\infty} (x-y) e^{-\frac{n^2y^2}{2}}\,dy. \end{aligned} $$
Let each term be $(\ast)$ and $(\ast\ast)$ respectively. Then
$$ \begin{aligned} (\ast) &= \frac{nx}{\sqrt{2\pi}} \int_{-\infty}^x e^{-\frac{n^2y^2}{2}} \,dy - \frac{n}{\sqrt{2\pi}} \int_{-\infty}^x ye^{-\frac{n^2y^2}{2}} \,dy\\ & = \frac{nx}{\sqrt{2\pi}} \int_{-\infty}^{\frac{nx}{\sqrt{2}}} e^{-t^2} \frac{\sqrt{2}}{n} \,dt - \frac{n}{\sqrt{2\pi}} \left[-\frac{1}{n^2} e^{-\frac{n^2y^2}{2}}\right]^x_{-\infty} \\ &=\frac{x}{2} \cdot \frac{2}{\sqrt{\pi}} \int_{-\infty}^{\frac{nx}{\sqrt{2}}} e^{-t^2} \,dt + \frac{n}{\sqrt{2\pi}}\cdot\frac{1}{n^2}e^{-\frac{n^2x^2}{2}} \\ &= \frac{x}{2} \left[\mathrm{erf}\left(\frac{nx}{\sqrt{2}} \right) + 1\right] + \frac{1}{n}\cdot \frac{1}{\sqrt{2\pi}} e^{-\frac{n^2x^2}{2}} \end{aligned} $$
I substituted $\frac{ny}{\sqrt{2}} = t$ in the second line. The second part is pretty much the same calculation, with the additional $\alpha$ multiplied.
$$ \begin{aligned} (\ast\ast) &= \alpha\frac{nx}{\sqrt{2\pi}} \int_x^{\infty} e^{-\frac{n^2y^2}{2}} \,dy - \alpha \frac{n}{\sqrt{2\pi}} \int_x^{\infty} ye^{-\frac{n^2y^2}{2}} \,dy \\ &= \alpha \frac{nx}{\sqrt{2\pi}} \int_{\frac{nx}{\sqrt{2}}}^{\infty} e^{-t^2} \frac{\sqrt{2}}{n} \,dt - \alpha \frac{n}{\sqrt{2\pi}} \left[-\frac{1}{n^2} e^{-\frac{n^2y^2}{2}}\right]_x^{\infty} \\ &=\frac{\alpha x}{2} \cdot \frac{2}{\sqrt{\pi}} \int^{\infty}_{\frac{nx}{\sqrt{2}}} e^{-t^2} \,dt - \frac{\alpha n}{\sqrt{2\pi}}\cdot\frac{1}{n^2}e^{-\frac{n^2x^2}{2}} \\ &=\frac{\alpha x}{2} \left[1 - \mathrm{erf}\left(\frac{nx}{\sqrt{2}}\right)\right] - \frac{\alpha}{n}\cdot \frac{1}{\sqrt{2\pi}}e^{-\frac{n^2x^2}{2}}. \end{aligned} $$
I also substituted $\frac{ny}{\sqrt{2}} = t$ in the second line. Adding the two results gives
$$ \begin{aligned} G(x, \alpha, n) =&\, \frac{x}{2}\left[\mathrm{erf}\left(\frac{nx}{\sqrt{2}}\right) + 1\right] + \frac{1}{n}\cdot \frac{1}{\sqrt{2\pi}}e^{-\frac{n^2x^2}{2}} \\ &+ \frac{\alpha x}{2}\left[1 - \mathrm{erf}\left(\frac{nx}{\sqrt{2}}\right)\right] - \frac{\alpha}{n}\cdot\frac{1}{\sqrt{2\pi}}e^{-\frac{n^2x^2}{2}} \\ =& \frac{1+\alpha}{2}x + \frac{1-\alpha}{2}x\,\mathrm{erf}\left(\frac{nx}{\sqrt{2}}\right) + \frac{1}{n}\cdot\frac{1}{\sqrt{2\pi}}e^{-\frac{n^2x^2}{2}} - \frac{\alpha}{n}\cdot\frac{1}{\sqrt{2\pi}}e^{-\frac{n^2x^2}{2}} \end{aligned} $$
My problem here is that the first 3 terms coincide with the result in the paper. But I have no idea where the last term came from.
The equations are tedious, so please understand my abuse of notation, and there may be some typos.
Thanks in advance for check this tedious calculation for me.
You’re right; the paper is wrong. This is easiest to see if you choose $\alpha=1$. Then the convolution of $x$ with a Gaussian should indeed yield $x$, not an additional Gaussian term. You don’t need to evaluate any integrals for that; it follows by symmetry, since a convolution of a symmetric and an antisymmetric function must yield an antisymmetric function.