Floating point arithmetic, again [Software]

posted by Helmut Homepage – Vienna, Austria, 2020-12-10 19:12 (129 d 22:28 ago) – Posting: # 22126
Views: 2,350

Dear Ohlbe,

» A colleague recently pointed to a strange behaviour in Excel (I have 2013, he has 2016, both give the same result here.

And I have 2000 (don’t laugh).

» Type =0.5-0.4-0.1: you'll get the expected result, 0.


» Now type =(0.5-0.4-0.1): you'll get -2,77556E-17. What ? Ain't that supposed to be the same calculation ?

In principle, yes. Unless we could inspect the source code by M$, difficult to tell what happens with the brackets.

» Trying =0=0.5-0.4-0.1: the answer I got was FALSE. WTF, if the answer you get is 0 ? It's not a matter of number of decimals displayed: I tried to add more to the first result, or to switch to scientific notation, but still got 0.

OK, write in A1 =0.5-0.4-0.1 and in B1 =A1=0
  1. What do you expect?
  2. What do you get?
[image] Floating point arithmetic, again (see also this post above).
Try in A1 =16-8-4-2-2, in B1 =(16-8-4-2-2), and in C1 =A1=B1. I bet it “works” as expected, since these numbers can be converted to binary digits without error.

What about [image]?

options("digits" = 16)
a <- 0.5-0.4-0.1
b <- (0.5-0.4-0.1)
a; b
[1] -2.775557561562891e-17
[1] -2.775557561562891e-17

identical(a, b)
[1] TRUE
c <- 16-8-4-2-2
d <- (16-8-4-2-2)
c; d
[1] 0
[1] 0

identical(c, d)
[1] TRUE

Dif-tor heh smusma 🖖
Helmut Schütz

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

Complete thread:

 Admin contact
21,419 posts in 4,475 threads, 1,510 registered users;
online 11 (0 registered, 11 guests [including 2 identified bots]).
Forum time: Monday 18:40 CEST (Europe/Vienna)

There is one certainty in drug development
and statistics that one can depend on:
the data are always late.    Scott Patterson and Byron Jones

The Bioequivalence and Bioavailability Forum is hosted by
BEBAC Ing. Helmut Schütz