precision in calculators

Question

The (square root of 0.99) minus 1 is

-0.005012562893380035 according to Java 8 on 64-bit Linux.

-0.005012562893380040 according to LibreOffice on 64-bit Linux.

-0.005012562 according to the Sharp EL-W535 calculator.

This is disappointing because at the 1 billionth decimal place the calculator is wrong because it should be a 3 (given that ...28 should round to ...3).

I bought this calculator because it has a very wide display and the LCD itself is capable of displaying more digits but unfortunately it only showed billionths. I can live with that but what I dislike that the digit is actually wrong. What accounts for this? Are such machines typically 32 bit or is there some other explanation? Even if it were a 32 bit machine, that is a poor excuse because it should be possible for even an 8 bit machine to provide the same answer as a 64 bit machine (albeit with more complicated software).

Answer 1

You can change the range outside which numbers are displayed in scientific notation, and then, it would most likely be something like -5.012562893E-3 in this case. Set up with J 1 4, according to the point "SET UP Menu" in the user manual.

Answer 2

You want to compute $\sqrt{1+x}-1$ with small $x$ (in your example $x=-0.01$). The direct computation is unstable because of catastrophic cancellation (think about $x=10^{-30}$ or so)! A stable version for small $x$ is $$\sqrt{1+x}-1 = \frac{x}{\sqrt{1+x}+1}$$