Floating point arithmetic, again [Software]

posted by Ohlbe – France, 2020-12-10 20:18 (129 d 20:05 ago) – Posting: # 22129
Views: 2,341

Hi ElMaestro,

» But things are tricky for a figure like 0.1 (one tenth) - in binary you'd probably represent it as .00011001100110011001100110011001100110011001100110011... where the 0011 blocks repeat indefinitely.
» In computers are data types have a maximum storage space, like 4 bytes or 8 bytes. So somewhere along the repeating 0011 blocks the computer will truncate the figure so it fits in the memory. And that is the source of your rounding issue.

Mmmm. But then if the problem comes from the nasty 0.1, how comes that if I type =(0.8-0.4-0.2-0.1-0.1) I get a nice 0, without any funky E-17 ? And then if I type =0=(0.8-0.4-0.2-0.1-0.1), or =0=0.8-0.4-0.2-0.1-0.1 I get a nice TRUE ?


Complete thread:

 Admin contact
21,419 posts in 4,475 threads, 1,510 registered users;
online 11 (0 registered, 11 guests [including 3 identified bots]).
Forum time: Monday 17:24 CEST (Europe/Vienna)

There is one certainty in drug development
and statistics that one can depend on:
the data are always late.    Scott Patterson and Byron Jones

The Bioequivalence and Bioavailability Forum is hosted by
BEBAC Ing. Helmut Schütz