Floating point arithmetic, again [Software]

posted by ElMaestro  – Denmark, 2020-12-10 21:38 (44 d 16:30 ago) – Posting: # 22130
Views: 1,132

Hi Ohlbe,

» Mmmm. But then if the problem comes from the nasty 0.1, how comes that if I type =(0.8-0.4-0.2-0.1-0.1) I get a nice 0, without any funky E-17 ? And then if I type =0=(0.8-0.4-0.2-0.1-0.1), or =0=0.8-0.4-0.2-0.1-0.1 I get a nice TRUE ?

Impossible to tell if this is an effect from Excel or from some degree of streamlining within the CPU.
It is fairly trivial (I think) at the CPU level as well as in Excel to check if you are handing objects of the type (x-x) and to substitute those with a zero where they occur.

Pass or fail!

Complete thread:

 Admin contact
21,310 posts in 4,445 threads, 1,489 registered users;
online 4 (0 registered, 4 guests [including 1 identified bots]).
Forum time: Sunday 14:09 CET (Europe/Vienna)

Every man gets a narrower and narrower field of knowledge
in which he must be an expert in order to compete with other people.
The specialist knows more and more about less and less
and finally knows everything about nothing.    Konrad Lorenz

The Bioequivalence and Bioavailability Forum is hosted by
BEBAC Ing. Helmut Schütz