Floating point arithmetic, again [Software]

posted by Ohlbe – France, 2020-12-10 19:13 (76 d 06:22 ago) – Posting: # 22128
Views: 1,618

Dear Helmut,

» OK, write in A1 =0.5-0.4-0.1 and in B1 =A1=0
»    1. What do you expect?
I expect A1 to read 0. Regarding B1: I would naturally expect it to return TRUE. But on the other hand, typing =0=0.5-0.4-0.1 returns FALSE, and I would not necessarily expect a two-step calculation to yield a result that differs from a one-step calculation. By the way I checked: =0=5-4-1 returns TRUE, so it is not that Excel just compares 0 to 0.5 and ignores the subsequent signs and figures.

»    2. What do you get?
A1 returns 0, as expected. Switching to scientific notation gives 0,00E+00. So it doesn't look like we have some hidden decimals there that don't show because of the default display.

B1 returns TRUE. Indeed the result I would normally expect. But this means that typing the comparison and the calculation in one cell, or the calculation in one cell and the comparison in another, gives different results.

» Try in A1 =16-8-4-2-2, in B1 =(16-8-4-2-2), and in C1 =A1=B1. I bet it “works” as expected, since these numbers can be converted to binary digits without error.
Yes. But it works also with A1 =5-4-1, in B1 =(5-4-1), and in C1 =A1=B1. So it works with 5, which is not a power of 2 ?
—
Regards
Ohlbe

Complete thread:

Activity
 Admin contact
21,355 posts in 4,458 threads, 1,493 registered users;
online 6 (0 registered, 6 guests [including 3 identified bots]).
Forum time: Thursday 01:35 UTC (Europe/Vienna)

The rise of biometry in this 20th century,
like that of geometry in the 3rd century before Christ,
seems to mark out one of the great ages or critical periods
in the advance of the human understanding.    R.A. Fisher

The Bioequivalence and Bioavailability Forum is hosted by
BEBAC Ing. Helmut Schütz
HTML5