Floating point arithmetic, again [Software]

posted by Helmut Homepage – Vienna, Austria, 2020-12-10 20:12 (1223 d 19:35 ago) – Posting: # 22126
Views: 8,770

Dear Ohlbe,

❝ A colleague recently pointed to a strange behaviour in Excel (I have 2013, he has 2016, both give the same result here.


And I have 2000 (don’t laugh).

❝ Type =0.5-0.4-0.1: you'll get the expected result, 0.


Yep.

❝ Now type =(0.5-0.4-0.1): you'll get -2,77556E-17. What ? Ain't that supposed to be the same calculation ?


In principle, yes. Unless we could inspect the source code by M$, difficult to tell what happens with the brackets.

❝ Trying =0=0.5-0.4-0.1: the answer I got was FALSE. WTF, if the answer you get is 0 ? It's not a matter of number of decimals displayed: I tried to add more to the first result, or to switch to scientific notation, but still got 0.


OK, write in A1 =0.5-0.4-0.1 and in B1 =A1=0
  1. What do you expect?
  2. What do you get?
Floating point arithmetic, again (see also this post above).
Try in A1 =16-8-4-2-2, in B1 =(16-8-4-2-2), and in C1 =A1=B1. I bet it “works” as expected, since these numbers can be converted to binary digits without error.

What about [image]?

options("digits" = 16)
a <- 0.5-0.4-0.1
b <- (0.5-0.4-0.1)
a; b
[1] -2.775557561562891e-17
[1] -2.775557561562891e-17

identical(a, b)
[1] TRUE
c <- 16-8-4-2-2
d <- (16-8-4-2-2)
c; d
[1] 0
[1] 0

identical(c, d)
[1] TRUE


Dif-tor heh smusma 🖖🏼 Довге життя Україна! [image]
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

Complete thread:

UA Flag
Activity
 Admin contact
22,985 posts in 4,823 threads, 1,653 registered users;
49 visitors (0 registered, 49 guests [including 7 identified bots]).
Forum time: 16:47 CEST (Europe/Vienna)

Complex, statistically improbable things are by their nature
more difficult to explain than
simple, statistically probable things.    Richard Dawkins

The Bioequivalence and Bioavailability Forum is hosted by
BEBAC Ing. Helmut Schütz
HTML5