Floating point arithmetic, again [Software]

posted by Helmut Homepage – Vienna, Austria, 2020-12-10 20:12 (724 d 20:33 ago) – Posting: # 22126
Views: 6,637

Dear Ohlbe,

❝ A colleague recently pointed to a strange behaviour in Excel (I have 2013, he has 2016, both give the same result here.


And I have 2000 (don’t laugh).

❝ Type =0.5-0.4-0.1: you'll get the expected result, 0.


Yep.

❝ Now type =(0.5-0.4-0.1): you'll get -2,77556E-17. What ? Ain't that supposed to be the same calculation ?


In principle, yes. Unless we could inspect the source code by M$, difficult to tell what happens with the brackets.

❝ Trying =0=0.5-0.4-0.1: the answer I got was FALSE. WTF, if the answer you get is 0 ? It's not a matter of number of decimals displayed: I tried to add more to the first result, or to switch to scientific notation, but still got 0.


OK, write in A1 =0.5-0.4-0.1 and in B1 =A1=0
  1. What do you expect?
  2. What do you get?
Floating point arithmetic, again (see also this post above).
Try in A1 =16-8-4-2-2, in B1 =(16-8-4-2-2), and in C1 =A1=B1. I bet it “works” as expected, since these numbers can be converted to binary digits without error.

What about [image]?

options("digits" = 16)
a <- 0.5-0.4-0.1
b <- (0.5-0.4-0.1)
a; b
[1] -2.775557561562891e-17
[1] -2.775557561562891e-17

identical(a, b)
[1] TRUE
c <- 16-8-4-2-2
d <- (16-8-4-2-2)
c; d
[1] 0
[1] 0

identical(c, d)
[1] TRUE


Dif-tor heh smusma 🖖🏼 Довге життя Україна! [image]
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

Complete thread:

UA Flag
Activity
 Admin contact
22,428 posts in 4,694 threads, 1,598 registered users;
16 visitors (0 registered, 16 guests [including 7 identified bots]).
Forum time: 16:45 CET (Europe/Vienna)

Statistics is the art of never having to say you’re wrong.
Variance is what any two statisticians are at.    C.J. Bradfield

The Bioequivalence and Bioavailability Forum is hosted by
BEBAC Ing. Helmut Schütz
HTML5