Rescaling, or finding logarithmic equivalent of exponential functions

Question

I'm working on an algorithm that gets the weighted least squares of some data and outputs really big, or really small numbers. I'm talking about numbers like $3.55114473577E+256$.

Now, part of the algorithm I'm coding needs to use these numbers and raise $e$ to their power, in the following formula:

$β = e^x / (1 + e^x) $

Where β values will then be used as weights for regression and such.

But $e^{3.55114473577e+256}$ is pretty impossible.

So in such problems, what's the best solution? What kind of rescaling should be done to the numbers I'm getting? Or, alternatively, what's the logarithmic equivalent to the above equation? Because I'm aware that to deal with gigantic numbers, logarithms can be used instead of exponents.

Answer 1

For large $x$, $\frac{e^x}{1+e^x}$ is very close to $1$. You have $$ 1 - \frac{e^x}{1 + e^x} = \frac{1}{1 + e^x} $$ so the relative error of assuming $\frac{e^x}{1+e^x} = 1$ is about $e^x$. That is so insanely tiny for $x \approx 10^{256}$ that I very much doubt that it matters. If your algorithm really depends on $\beta$, i.e. if you cannot just replace it with $1$, then I doubt that what you're trying to do is numerically feasable.

---

More generally, the function $f(x) = \frac{e^x}{1 + e^x}$ has the following behaviour

1. $f$ is a strictly monotonic function from $\mathbb{R}$ to $(0,1)$.
2. $f(0) = \frac{1}{2}$, $f(x) \to 1$ as $x \to \infty$, $f(x) \to 0$ as $x \to -\infty$.
3. For $x \geq 10^n$, $f(x) \geq 1 - 10^{-\left(10^{n-1}\right)}$.
4. For $x \leq -10^n$, $f(x) \leq 10^{-\left(10^{n-1}\right)}$

(3) and (4) should give you an idea - depending on the precision of your numbers - how large $x$ must be to safely replace $\frac{e^x}{1 + e^x}$ with $1$.