show that $\log(1+e^z) = \log(1+e^{-|z|}) + \max(0, z)$ and explain why the right latter is more numerically stable

Question

I used 2 separate cases. First case is when $z < 0$ and second case $z \geq 0$. I proved the first case but I have no clue how to deal with the second.

prove that $\log(1+e^z) = \log(1+e^{-|z|}) + \max (z, 0)$ :

1. we use separate cases. first case, when $ z < 0 $

   • $\log(1+e^z)$
   • $\operatorname{sign}(z) = -1$ because $z < 0$ therefore $z = -|z|$
   • $\log(1+e^z) = \log(1+e^{-|z|})$
   • $z < 0$ therefore $\max (z, 0) = 0$
   • $\log(1+e^z) = \log(1+e^{-|z|}) + \max (z, 0)$

2. when $ z \geq 0$

   • $\max (z, 0) = z$
   • $\log(1+e^z) = \log(1+e^{-|z|}) + z$
   • $\log(1+e^z) - \log(1+e^{-|z|}) = z$
   • $\log(\frac{1+e^z}{1+e^{-|z|}}) = z$

am I on right path for the second case? how to prove the identity?

Answer 1

When $z \ge 0$, we have $$e^{-|z|} = e^{-z},$$ hence $$\log(1 + e^z) = \log(e^z e^{-z} + e^z) = \log \left(e^z (e^{-z} + 1)\right) = \log e^z + \log (1 + e^{-z}) = z + \log(1 + e^{-|z|}).$$

A crucial hint as to the reason why the expression $\log(1+e^{-|z|}) + \max(0,z)$ is more numerically stable than $\log(1+e^z)$ is seen in the $\max(0,z)$ term. When $z$ is positive and "large," $e^z$ will be extremely large; then adding $1$ will give you a number that is almost the same as $e^z$, and taking the logarithm will give you a number that is almost the same as $z$. But there is very little precision in this result because while you already knew that $\log(1 + e^z) \approx z$ when $z$ is large, you won't be able to preserve the error term with the naive computation.

Let us illustrate this with a numeric example. Suppose $z = 1000$, which by most computer programming languages is not even "large." Then the naive computation is $$\log(1 + e^{1000}).$$ You will note that $e^{1000} \approx 10^{434}$. Adding $1$ and computing the logarithm will give the result $$1000.00000000000000000$$ in most programming languages. But in fact, you can get a far more precise result, which is $$1000 + \log(1 + e^{-1000}) \approx 1000 + e^{-1000} \approx 1000 + 5.076 \times 10^{-435}.$$ The approximation that was used here is the series $$\log (1+z) = z - \frac{z^2}{2} + \frac{z^3}{3} - \ldots, \quad |z| < 1,$$ which converges most rapidly when $|z|$ is close to $0$. As this is certainly the case for $z = e^{-1000}$, with just one term of this series expansion we now have well over $400$ decimal digits of precision. This result simply is not attainable through the naive calculation. In some programming languages, for sufficiently large $z$, the floating-point calculation of $\log(1 + e^z)$ will result in overflow because $e^z$ will be too large to express; e.g., select $z = 10^{100}$.

Answer 2

For the second case, you should notice that when $z \geq 0$ you have $|z| = z$ therefore, $$ \log\left(\dfrac{1+e^{z}}{1+e^{-|z|}} \right) = \log\left(\dfrac{1+e^{z}}{1+e^{-z}} \right) = \log\left(\dfrac{e^z(e^{-z}+1)}{1+e^{-z}} \right) = \log(e^{z})=z = \max(z,0) $$

Answer 3

You should take a look at wiki logsumexp In your expression, we should recognize the logarithm of the sum of two exponentials with $1=e^0$ so this is a particular case of logsumexp trick.

Let $m=\max(0,x)$ and write \begin{eqnarray*} \log \left( 1+e^x \right) &=& \log \left[ e^m \left(e^{-m}+e^{x-m}\right) \right] \\ &=& m + \log \left(e^{-m}+e^{x-m} \right) \end{eqnarray*} If $x<0$, the log argument writes $\left(1+e^{x} \right)$. If $x>0$, it writes $\left(e^{-x}+1 \right)$. So a general expression is $\left(1+e^{-|x|} \right)$. Finally, we can write \begin{eqnarray*} \log \left( 1+e^x \right) &=& m + \log \left(1+e^{-|x|} \right) \end{eqnarray*}