Floating point arithmetic, again [Software]

posted by Ohlbe – France, 2020-12-10 21:13 (1505 d 14:28 ago) – Posting: # 22128
Views: 17,974

Dear Helmut,

❝ OK, write in A1 =0.5-0.4-0.1 and in B1 =A1=0


❝    1. What do you expect?


I expect A1 to read 0. Regarding B1: I would naturally expect it to return TRUE. But on the other hand, typing =0=0.5-0.4-0.1 returns FALSE, and I would not necessarily expect a two-step calculation to yield a result that differs from a one-step calculation. By the way I checked: =0=5-4-1 returns TRUE, so it is not that Excel just compares 0 to 0.5 and ignores the subsequent signs and figures.

❝    2. What do you get?


A1 returns 0, as expected. Switching to scientific notation gives 0,00E+00. So it doesn't look like we have some hidden decimals there that don't show because of the default display.

B1 returns TRUE. Indeed the result I would normally expect. But this means that typing the comparison and the calculation in one cell, or the calculation in one cell and the comparison in another, gives different results.

❝ Try in A1 =16-8-4-2-2, in B1 =(16-8-4-2-2), and in C1 =A1=B1. I bet it “works” as expected, since these numbers can be converted to binary digits without error.


Yes. But it works also with A1 =5-4-1, in B1 =(5-4-1), and in C1 =A1=B1. So it works with 5, which is not a power of 2?
—
Regards
Ohlbe

Complete thread:

UA Flag
Activity
 Admin contact
23,368 posts in 4,908 threads, 1,672 registered users;
144 visitors (0 registered, 144 guests [including 12 identified bots]).
Forum time: 11:42 CET (Europe/Vienna)

The mediocre teacher tells.
The good teacher explains.
The superior teacher demonstrates.
The great teacher inspires.    William Arthur Ward

The Bioequivalence and Bioavailability Forum is hosted by
BEBAC Ing. Helmut Schütz
HTML5