Rounding trouble in a calculator's square root function

Question

I made the following observation following a suggestion from a friend. This question is about the Microsoft calculator program implemented on Windows. Since I don't have access to any other calculator at the moment, I cannot confirm whether this is a general feature.

If you enter an integer in the calculator program, say $N$, hit the minus button followed by the same integer again, you get zero as expected.

However, if you input $N^2$, then hit square root button, you get $N$, then subtract $N$, you get a very small number, but not zero.

This has been my observation for several integers that I have tried.

For example, with $N=5$, you get $1.232\times 10^{-24}$ in the final step.

Now, I understand computers use finite precision (and hence the square root function must be working under finite precision). However, I don't understand why this shows up in the final step only? Is it only because this is a small number in comparison to any integer, but when you get $0$, you can no longer make that approximation?

Answer 1

You'll find your answer and much more in the articles below:

What Every Computer Scientist Should Know About Floating-Point Arithmetic

Also see this article on Calculator Forensics, esp. "Hints and Tricks For Obtaining Guard Digits".

Answer 2

J.M., since when is 24 decimal digits accuracy considered a bad approximation?

Original poster, they could show sqrt(5)**2 as

5.000000000000000000000001232

but since the display probably has fewer display digits than this, they chop it to "5" as that is the closest representation in the allowed number of digits.

BTW, I just tried this in "calc" in windows 7 and the 5-sqrt(5)**2 is 1.7599e-37

If you want exact math, you will need to use a symbolic math package.