Floating point arithmetic, again [Software]

posted by Ohlbe – France, 2020-12-10 21:18 (722 d 02:29 ago) – Posting: # 22129
Views: 6,641

Hi ElMaestro,

❝ But things are tricky for a figure like 0.1 (one tenth) - in binary you'd probably represent it as .00011001100110011001100110011001100110011001100110011... where the 0011 blocks repeat indefinitely.

❝ In computers are data types have a maximum storage space, like 4 bytes or 8 bytes. So somewhere along the repeating 0011 blocks the computer will truncate the figure so it fits in the memory. And that is the source of your rounding issue.


Mmmm. But then if the problem comes from the nasty 0.1, how comes that if I type =(0.8-0.4-0.2-0.1-0.1) I get a nice 0, without any funky E-17 ? And then if I type =0=(0.8-0.4-0.2-0.1-0.1), or =0=0.8-0.4-0.2-0.1-0.1 I get a nice TRUE ?

Regards
Ohlbe

Complete thread:

UA Flag
Activity
 Admin contact
22,424 posts in 4,694 threads, 1,600 registered users;
20 visitors (2 registered, 18 guests [including 7 identified bots]).
Forum time: 23:48 CET (Europe/Vienna)

The rise of biometry in this 20th century,
like that of geometry in the 3rd century before Christ,
seems to mark out one of the great ages or critical periods
in the advance of the human understanding.    R.A. Fisher

The Bioequivalence and Bioavailability Forum is hosted by
BEBAC Ing. Helmut Schütz
HTML5