## Floating point arithmetic, again [Software]

Dear Ohlbe,

» A colleague recently pointed to a strange behaviour in Excel (I have 2013, he has 2016, both give the same result here.

And I have 2000 (don’t laugh).

» Type =0.5-0.4-0.1: you'll get the expected result, 0.

Yep.

» Now type =(0.5-0.4-0.1): you'll get -2,77556E-17. What ? Ain't that supposed to be the same calculation ?

In principle, yes. Unless we could inspect the source code by M\$, difficult to tell what happens with the brackets.

» Trying =0=0.5-0.4-0.1: the answer I got was FALSE. WTF, if the answer you get is 0 ? It's not a matter of number of decimals displayed: I tried to add more to the first result, or to switch to scientific notation, but still got 0.

OK, write in =0.5-0.4-0.1 and in =A1=0
1. What do you expect?
2. What do you get?
Try in =16-8-4-2-2, in =(16-8-4-2-2), and in =A1=B1. I bet it “works” as expected, since these numbers can be converted to binary digits without error.

options("digits" = 16) a <- 0.5-0.4-0.1 b <- (0.5-0.4-0.1) a; b [1] -2.775557561562891e-17 [1] -2.775557561562891e-17 identical(a, b) [1] TRUE c <- 16-8-4-2-2 d <- (16-8-4-2-2) c; d [1] 0 [1] 0 identical(c, d) [1] TRUE

Dif-tor heh smusma 🖖
Helmut Schütz

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

22,181 posts in 4,647 threads, 1,572 registered users;
online 5 (0 registered, 5 guests [including 5 identified bots]).
Forum time: Saturday 17:22 CEST (Europe/Vienna)

Reach for the stars,
even if you have to stand on a cactus.    Susan Longacre

The Bioequivalence and Bioavailability Forum is hosted by
Ing. Helmut Schütz