Floating point arithmetic, again [Software]

posted by Ohlbe – France, 2020-12-10 21:13 (2001 d 18:42 ago) – Posting: # 22128
Views: 28,168

Dear Helmut,

❝ OK, write in A1 =0.5-0.4-0.1 and in B1 =A1=0


❝    1. What do you expect?


I expect A1 to read 0. Regarding B1: I would naturally expect it to return TRUE. But on the other hand, typing =0=0.5-0.4-0.1 returns FALSE, and I would not necessarily expect a two-step calculation to yield a result that differs from a one-step calculation. By the way I checked: =0=5-4-1 returns TRUE, so it is not that Excel just compares 0 to 0.5 and ignores the subsequent signs and figures.

❝    2. What do you get?


A1 returns 0, as expected. Switching to scientific notation gives 0,00E+00. So it doesn't look like we have some hidden decimals there that don't show because of the default display.

B1 returns TRUE. Indeed the result I would normally expect. But this means that typing the comparison and the calculation in one cell, or the calculation in one cell and the comparison in another, gives different results.

❝ Try in A1 =16-8-4-2-2, in B1 =(16-8-4-2-2), and in C1 =A1=B1. I bet it “works” as expected, since these numbers can be converted to binary digits without error.


Yes. But it works also with A1 =5-4-1, in B1 =(5-4-1), and in C1 =A1=B1. So it works with 5, which is not a power of 2?
—
Regards
Ohlbe

Complete thread:

UA Flag
Activity
 Admin contact
23,653 posts in 4,991 threads, 1,570 registered users;
114 visitors (0 registered, 114 guests [including 16 identified bots]).
Forum time: 16:55 CEST (Europe/Vienna)

I’m all in favor of the democratic principle
that one idiot is as good as one genius, but I draw the line
when someone takes the next step and concludes
that two idiots are better than one genius.    Leo Szilard

The Bioequivalence and Bioavailability Forum is hosted by
BEBAC Ing. Helmut Schütz
HTML5