Floating point arithmetic, again [Software]

posted by ElMaestro  – Denmark, 2020-12-10 20:40 (1455 d 05:19 ago) – Posting: # 22127
Views: 17,278

Hi Ohlbe,

❝ Type =0.5-0.4-0.1: you'll get the expected result, 0.


❝ Now type =(0.5-0.4-0.1): you'll get -2,77556E-17. What ? Ain't that supposed to be the same calculation?


Yes it is (or may be) to you and to me, but that isn't how an electronic brain works.

❝ Trying =0=0.5-0.4-0.1: the answer I got was FALSE. WTF, if the answer you get is 0 ? It's not a matter of number of decimals displayed: I tried to add more to the first result, or to switch to scientific notation, but still got 0.


In the world of 1's and 0's you cannot represent all decimal numbers that are easily displayed in our common number format using digits of 0 to 9 ("decimal"). Your little equations exemplifies it wonderfully; it is not per se an Excel issue.

Here's R:

0==(0.5-0.4-0.1)
[1] FALSE


You can totally represent .5 (a half) in binary: It is actually .1 because it is 2 raised to the power of -1.
.11 in binary would be 2 raised to the power of -1 plus 2 raised to the power of -2. And so on.

But things are tricky for a figure like 0.1 (one tenth) - in binary you'd probably represent it as .00011001100110011001100110011001100110011001100110011... where the 0011 blocks repeat indefinitely.
In computers are data types have a maximum storage space, like 4 bytes or 8 bytes. So somewhere along the repeating 0011 blocks the computer will truncate the figure so it fits in the memory. And that is the source of your rounding issue.

Computer scientists know never to compare fractional numbers uncritically.
It stinks. The problem is not Excel, but that we have invested computers that use binary when at the same time we like to think of numbers represented with the usual 10 digits ("decimal"). :-)

Pass or fail!
ElMaestro

Complete thread:

UA Flag
Activity
 Admin contact
23,332 posts in 4,899 threads, 1,660 registered users;
23 visitors (0 registered, 23 guests [including 7 identified bots]).
Forum time: 01:59 CET (Europe/Vienna)

I don’t write drafts.
I write from the beginning to the end,
and when it’s finished, it’s done.    Clifford Geertz

The Bioequivalence and Bioavailability Forum is hosted by
BEBAC Ing. Helmut Schütz
HTML5