Floating point arithmetic, again [Software]

posted by Ohlbe – France, 2020-12-10 20:13 (655 d 19:55 ago) – Posting: # 22128
Views: 6,698

Dear Helmut,

» OK, write in A1 =0.5-0.4-0.1 and in B1 =A1=0

»    1. What do you expect?

I expect A1 to read 0. Regarding B1: I would naturally expect it to return TRUE. But on the other hand, typing =0=0.5-0.4-0.1 returns FALSE, and I would not necessarily expect a two-step calculation to yield a result that differs from a one-step calculation. By the way I checked: =0=5-4-1 returns TRUE, so it is not that Excel just compares 0 to 0.5 and ignores the subsequent signs and figures.

»    2. What do you get?

A1 returns 0, as expected. Switching to scientific notation gives 0,00E+00. So it doesn't look like we have some hidden decimals there that don't show because of the default display.

B1 returns TRUE. Indeed the result I would normally expect. But this means that typing the comparison and the calculation in one cell, or the calculation in one cell and the comparison in another, gives different results.

» Try in A1 =16-8-4-2-2, in B1 =(16-8-4-2-2), and in C1 =A1=B1. I bet it “works” as expected, since these numbers can be converted to binary digits without error.

Yes. But it works also with A1 =5-4-1, in B1 =(5-4-1), and in C1 =A1=B1. So it works with 5, which is not a power of 2 ?

Regards
Ohlbe

Complete thread:

UA Flag
Activity
 Admin contact
22,391 posts in 4,685 threads, 1,594 registered users;
online 23 (0 registered, 23 guests [including 22 identified bots]).
Forum time: Tuesday 17:09 CEST (Europe/Vienna)

Statistics. A sort of elementary form of mathematics which consists of
adding things together and occasionally squaring them.    Stephen Senn

The Bioequivalence and Bioavailability Forum is hosted by
BEBAC Ing. Helmut Schütz
HTML5