## To round or not to round… [Software]

» » Haha, I know this game and therefore, always round before the comparison.

»

» or never bring yourself into a situation where you need to compare floats.

Did I tell to how

*many*significant digits I round?

» I've taken the worst way out and written functions that check if x is simar to y plus/minus a wee faction like 1e-8 or something.

I don’t know which kind of measurements you are comparing. Sumfink ultra-precise (according to the International Bureau of Weights and Measures:

*t*±5×10

^{–16},

*l*±2.1×10

^{–11})?

Zizou (I guess) and I were talking about concentrations. AP according to the GLs 20% at the LLOQ and 15% above. Hence,

*everything*reported substantially beyond that is not relevant (shall I call it noise?). What substantially means, depends on the field. Some people go crazy with 6σ, physicists are fine with just 3σ.

`x <- pi`

p <- 15

s <- 8

mult <- c(3, 6)

sigma <- x * p / 100

sigma.m <- sigma * mult

lo <- x - sigma.m

hi <- x + sigma.m

df <- data.frame(x = rep(x, 2), prec = p,

sigma = sigma, mult = mult,

sigma.m = sigma.m,

lo = lo, hi = hi)

df <- signif(df, s)

names(df)[2] <- "prec. (%)"

print(df, row.names = FALSE)

x prec. (%) sigma mult sigma.m lo hi

3.141593 15 0.4712389 3 1.413717 1.7278760 4.555309

3.141593 15 0.4712389 6 2.827433 0.3141593 5.969026

^{1}or 3.141 592 653 589 793 in an electronic file. However, I insist in a CRC-checksum

^{2}to verify the data-transfer. If I deal with bloody Excel, I round to one decimal beyond what is given in the analytical report

^{3}being aware that it is

*far*beyond the analytical AP. If the data-transfer of analytical results to stats was done electronically in “full numeric precision” (haha), I want to see validation for it.

^{4}

» It is ugly and clumsy, it works, and it feel everytime like I am suffering defeat.

Well, you are the C-man here. What about

`printf("%.yg\n", x);`

where `y`

is the desired number of significant digits? With R’s ` signif()`

:`options("digits" = 15)`

x <- 12345678.999999

y <- 0.12345678999999

prec <- 8

fmt1 <- "%33.17f"

fmt2 <- paste0("%.", prec, "g")

cat(x, "\n",

y, "\n",

sprintf(fmt1, x), "\u2190 fake news\n",

sprintf(fmt1, y), "\u2190 fake news\n",

sprintf(fmt1, signif(x, prec)), "\u2190", prec, "significant digits\n",

sprintf(fmt1, signif(y, prec)), "\u2190", prec, "significant digits\n",

sprintf(fmt2, x), "\u2190", "directly with", paste0("'", fmt2, "'\n"),

sprintf(fmt2, y), "\u2190", "directly with", paste0("'", fmt2, "'\n"))

12345678.999999

0.12345678999999

12345678.99999899975955486 ← fake news

0.12345678999999000 ← fake news

12345679.00000000000000000 ← 8 significant digits

0.12345679000000000 ← 8 significant digits

12345679 ← directly with '%.8g'

0.12345679 ← directly with '%.8g'

If you are interested whether rubbish in ≈ rubbish out, ask for a checksum and verify it. Probably better than diving into the murky waters of (likely irrelevant) rounding.

- I would be fine with just 3.1 for π. The RE is –1.32%, again much below what is achievable in bioanalytics.

- SHA-256 or higher preferred (collisions reported for SHA-1,
*i.e.*, different input give the same hash). MD5 is better than nothing.

- That’s the only GxP-compliant (dated/signed, released by QUA) document, right? Did you ever see result-tables with 15 significant digits?

- Here it gets tricky. These data are
*different*to what is given in the analytical report. Now what? At least I expect a statement about this discrepancy in the protocols (analytical and/or statistical). Regularly I see something like “calculations were performed in full numeric precision”. How could one ever hope to verify that having only the analytical report with rounded results?

Cheers,

Helmut Schütz

The quality of responses received is directly proportional to the quality of the question asked. 🚮

Science Quotes

### Complete thread:

- Spreadsheet failures, any recent examples? ElMaestro 2019-07-18 13:43 [Software]
- Spreadsheet failures, any recent examples? Ohlbe 2019-07-18 14:32
- Spreadsheet addiction Helmut 2019-07-18 16:12
- Nasty beast Ohlbe 2019-07-18 17:30
- Nasty beast Helmut 2019-07-18 20:26
- Nasty beast Ohlbe 2019-07-19 11:32
- Nasty beast ElMaestro 2019-07-19 12:32
- Decidedly off topic Ohlbe 2019-07-19 13:38

- OT: R limbo 101 Helmut 2019-07-19 13:00

- Nasty beast ElMaestro 2019-07-19 12:32

- Nasty beast Ohlbe 2019-07-19 11:32

- Nasty beast Helmut 2019-07-18 20:26
- Spreadsheet addiction Shuanghe 2019-07-18 18:59
- Spreadsheet addiction Helmut 2019-07-18 20:04
- OT: Spreadsheet addiction Shuanghe 2019-07-19 12:41
- OT: Spreadsheet addiction Helmut 2019-07-19 14:29
- OT: Spreadsheet addiction nobody 2019-07-19 15:53
- OT: Spreadsheet addiction Helmut 2019-07-19 19:32

- OT: Spreadsheet addiction nobody 2019-07-19 15:53

- OT: Spreadsheet addiction Helmut 2019-07-19 14:29

- OT: Spreadsheet addiction Shuanghe 2019-07-19 12:41

- Spreadsheet addiction Helmut 2019-07-18 20:04

- Nasty beast Ohlbe 2019-07-18 17:30
- Spreadsheet etc. failures zizou 2019-07-20 00:18
- As designed ☺ Helmut 2019-07-20 02:12
- As designed ☺ ElMaestro 2019-07-20 09:08
- To round or not to round…Helmut 2019-07-20 12:53
- To round or not to round… ElMaestro 2019-07-20 19:59
- floating-point math is always more complex than you think it is mittyri 2019-07-20 22:43

- To round or not to round… ElMaestro 2019-07-20 19:59

- To round or not to round…Helmut 2019-07-20 12:53

- As designed ☺ ElMaestro 2019-07-20 09:08

- As designed ☺ Helmut 2019-07-20 02:12

- Spreadsheet addiction Helmut 2019-07-18 16:12

- Spreadsheet failures, any recent examples? Ohlbe 2019-07-18 14:32