Floating point arithmetic, again [Software]

posted by Helmut Homepage – Vienna, Austria, 2020-12-10 20:12 (1174 d 15:10 ago) – Posting: # 22126
Views: 8,288

Dear Ohlbe,

❝ A colleague recently pointed to a strange behaviour in Excel (I have 2013, he has 2016, both give the same result here.


And I have 2000 (don’t laugh).

❝ Type =0.5-0.4-0.1: you'll get the expected result, 0.


Yep.

❝ Now type =(0.5-0.4-0.1): you'll get -2,77556E-17. What ? Ain't that supposed to be the same calculation ?


In principle, yes. Unless we could inspect the source code by M$, difficult to tell what happens with the brackets.

❝ Trying =0=0.5-0.4-0.1: the answer I got was FALSE. WTF, if the answer you get is 0 ? It's not a matter of number of decimals displayed: I tried to add more to the first result, or to switch to scientific notation, but still got 0.


OK, write in A1 =0.5-0.4-0.1 and in B1 =A1=0
  1. What do you expect?
  2. What do you get?
Floating point arithmetic, again (see also this post above).
Try in A1 =16-8-4-2-2, in B1 =(16-8-4-2-2), and in C1 =A1=B1. I bet it “works” as expected, since these numbers can be converted to binary digits without error.

What about [image]?

options("digits" = 16)
a <- 0.5-0.4-0.1
b <- (0.5-0.4-0.1)
a; b
[1] -2.775557561562891e-17
[1] -2.775557561562891e-17

identical(a, b)
[1] TRUE
c <- 16-8-4-2-2
d <- (16-8-4-2-2)
c; d
[1] 0
[1] 0

identical(c, d)
[1] TRUE


Dif-tor heh smusma 🖖🏼 Довге життя Україна! [image]
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

Complete thread:

UA Flag
Activity
 Admin contact
22,912 posts in 4,806 threads, 1,637 registered users;
28 visitors (0 registered, 28 guests [including 6 identified bots]).
Forum time: 11:22 CET (Europe/Vienna)

Exploratory analysis: The art of finding a Rembrandt
in a Jackson Pollock.    Stephen Senn

The Bioequivalence and Bioavailability Forum is hosted by
BEBAC Ing. Helmut Schütz
HTML5