Floating point arithmetic, again [Software]

posted by Ohlbe – France, 2020-12-10 21:13 (1289 d 04:33 ago) – Posting: # 22128
Views: 9,932

Dear Helmut,

❝ OK, write in A1 =0.5-0.4-0.1 and in B1 =A1=0


❝    1. What do you expect?


I expect A1 to read 0. Regarding B1: I would naturally expect it to return TRUE. But on the other hand, typing =0=0.5-0.4-0.1 returns FALSE, and I would not necessarily expect a two-step calculation to yield a result that differs from a one-step calculation. By the way I checked: =0=5-4-1 returns TRUE, so it is not that Excel just compares 0 to 0.5 and ignores the subsequent signs and figures.

❝    2. What do you get?


A1 returns 0, as expected. Switching to scientific notation gives 0,00E+00. So it doesn't look like we have some hidden decimals there that don't show because of the default display.

B1 returns TRUE. Indeed the result I would normally expect. But this means that typing the comparison and the calculation in one cell, or the calculation in one cell and the comparison in another, gives different results.

❝ Try in A1 =16-8-4-2-2, in B1 =(16-8-4-2-2), and in C1 =A1=B1. I bet it “works” as expected, since these numbers can be converted to binary digits without error.


Yes. But it works also with A1 =5-4-1, in B1 =(5-4-1), and in C1 =A1=B1. So it works with 5, which is not a power of 2 ?

Regards
Ohlbe

Complete thread:

UA Flag
Activity
 Admin contact
23,059 posts in 4,841 threads, 1,662 registered users;
36 visitors (0 registered, 36 guests [including 5 identified bots]).
Forum time: 02:46 CEST (Europe/Vienna)

There are only two kinds of scholars;
those who love ideas and those who hate them.    Emile Chartier

The Bioequivalence and Bioavailability Forum is hosted by
BEBAC Ing. Helmut Schütz
HTML5