Why sum of sigmoids is a good approximation of softplus function?

1.3k Views Asked by At

According to this paper: Rectified Linear Units Improve Restricted Boltzmann Machines,

$\sum_{i=1}^N \sigma(x-i+\tfrac{1}{2}) \approx \log(1+e^x)$ (equation 7)

where $\sigma(z) = \frac{1}{1+e^{-x}}$

Would anyone please explain me why it is so?

2

There are 2 best solutions below

1
On BEST ANSWER

The approximation was probably obtained thanks to two integrals encircling the sum (supposed with $N$ large enough), as shown in attachment :

enter image description here

In addition, the figure below shows the relative deviation as a function of $x$

enter image description here

0
On

The sum $\sum_{i=1}^{\infty} \sigma(x-i+1/2)$ is precisely the Riemann sum for the integral

$$\int_{1/2}^{\infty} \sigma(x-y+1/2)\,dy = \log(1+e^x)$$

using the midpoint rule with subintervals of length $\Delta x = 1$.