Bioequivalence and Bioavailability Forum • Floating point arithmetic, again

Floating point arithmetic, again [Software]

posted by Ohlbe – France, 2020-12-10 21:18 (1671 d 15:12 ago) – Posting: # 22129
Views: 19,997

Hi ElMaestro,

❝ But things are tricky for a figure like 0.1 (one tenth) - in binary you'd probably represent it as .00011001100110011001100110011001100110011001100110011... where the 0011 blocks repeat indefinitely.

❝ In computers are data types have a maximum storage space, like 4 bytes or 8 bytes. So somewhere along the repeating 0011 blocks the computer will truncate the figure so it fits in the memory. And that is the source of your rounding issue.

Mmmm. But then if the problem comes from the nasty 0.1, how comes that if I type =(0.8-0.4-0.2-0.1-0.1) I get a nice 0, without any funky E-17 ? And then if I type =0=(0.8-0.4-0.2-0.1-0.1), or =0=0.8-0.4-0.2-0.1-0.1 I get a nice TRUE ?

—
Regards
Ohlbe

Complete thread:

Spreadsheet failures, any recent examples? ElMaestro 2019-07-18 13:43
- Spreadsheet failures, any recent examples? Ohlbe 2019-07-18 14:32
  - Spreadsheet addiction Helmut 2019-07-18 16:12
    - Nasty beast Ohlbe 2019-07-18 17:30
      - Nasty beast Helmut 2019-07-18 20:26
        
        Nasty beast Ohlbe 2019-07-19 11:32
        
        Nasty beast ElMaestro 2019-07-19 12:32
        
        Decidedly off topic Ohlbe 2019-07-19 13:38
        
        OT: R limbo 101 Helmut 2019-07-19 13:00
    - Spreadsheet addiction Shuanghe 2019-07-18 18:59
      - Spreadsheet addiction Helmut 2019-07-18 20:04
        
        OT: Spreadsheet addiction Shuanghe 2019-07-19 12:41
        
        OT: Spreadsheet addiction Helmut 2019-07-19 14:29
        
        OT: Spreadsheet addiction nobody 2019-07-19 15:53
        
        OT: Spreadsheet addiction Helmut 2019-07-19 19:32
  - Spreadsheet etc. failures zizou 2019-07-20 00:18
    - As designed ☺ Helmut 2019-07-20 02:12
      - As designed ☺ ElMaestro 2019-07-20 09:08
        
        To round or not to round… Helmut 2019-07-20 12:53
        
        To round or not to round… ElMaestro 2019-07-20 19:59
        
        floating-point math is always more complex than you think it is mittyri 2019-07-20 22:43
- Spreadsheet failures, any recent examples? Ohlbe 2020-12-10 18:43
  - Floating point arithmetic, again Helmut 2020-12-10 19:12
    - Floating point arithmetic, again ElMaestro 2020-12-10 19:40
      - Floating point arithmetic, againOhlbe 2020-12-10 20:18
        
        Floating point arithmetic, again ElMaestro 2020-12-10 21:38
        
        Floating point arithmetic, again Ohlbe 2020-12-10 21:46
        
        Floating point arithmetic, again ElMaestro 2020-12-10 22:05
        
        Depressed Ohlbe 2020-12-10 23:35
        
        Depressed ElMaestro 2020-12-11 07:59
    - Floating point arithmetic, again Ohlbe 2020-12-10 20:13
  - From bad to worse Ohlbe 2020-12-10 22:11
    - From bad to worse mittyri 2020-12-11 00:22
    - All is good Helmut 2020-12-11 00:36
      - Float is float PharmCat 2020-12-18 20:53
        
        Float is float! Helmut 2020-12-20 23:27
        
        rational solution in R mittyri 2020-12-21 13:49
        
        related stuff Helmut 2021-01-14 12:53
        
        related stuff SDavis 2021-02-09 12:02
        
        related stuff ElMaestro 2021-02-09 19:55