Floating point arithmetic, again [Software]

posted by Helmut Homepage – Vienna, Austria, 2020-12-10 19:12 (85 d 10:07 ago) – Posting: # 22126
Views: 1,804

Dear Ohlbe,

» A colleague recently pointed to a strange behaviour in Excel (I have 2013, he has 2016, both give the same result here.

And I have 2000 (don’t laugh).

» Type =0.5-0.4-0.1: you'll get the expected result, 0.

Yep.

» Now type =(0.5-0.4-0.1): you'll get -2,77556E-17. What ? Ain't that supposed to be the same calculation ?

In principle, yes. Unless we could inspect the source code by M$, difficult to tell what happens with the brackets.

» Trying =0=0.5-0.4-0.1: the answer I got was FALSE. WTF, if the answer you get is 0 ? It's not a matter of number of decimals displayed: I tried to add more to the first result, or to switch to scientific notation, but still got 0.

OK, write in A1 =0.5-0.4-0.1 and in B1 =A1=0
  1. What do you expect?
  2. What do you get?
[image] Floating point arithmetic, again (see also this post above).
Try in A1 =16-8-4-2-2, in B1 =(16-8-4-2-2), and in C1 =A1=B1. I bet it “works” as expected, since these numbers can be converted to binary digits without error.

What about [image]?

options("digits" = 16)
a <- 0.5-0.4-0.1
b <- (0.5-0.4-0.1)
a; b
[1] -2.775557561562891e-17
[1] -2.775557561562891e-17

identical(a, b)
[1] TRUE
c <- 16-8-4-2-2
d <- (16-8-4-2-2)
c; d
[1] 0
[1] 0

identical(c, d)
[1] TRUE


Dif-tor heh smusma 🖖
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

Complete thread:

Activity
 Admin contact
21,370 posts in 4,463 threads, 1,495 registered users;
online 6 (0 registered, 6 guests [including 3 identified bots]).
Forum time: Saturday 05:19 CET (Europe/Vienna)

If debugging is the process of removing bugs,
then programming must be the process of putting them in.    Edsger W. Dijkstra

The Bioequivalence and Bioavailability Forum is hosted by
BEBAC Ing. Helmut Schütz
HTML5