Helmut
★★★

Vienna, Austria,
2021-02-19 12:02
(75 d 13:27 ago)

Posting: # 22214
Views: 983

## Probability to pass multiple studies [Power / Sample Size]

Dear all (and esp. our expert in probability zizou),

say, we have n studies, each powered at 90%. What is the probability (i.e., power) that all of them pass BE?

Let’s keep it simple: T/R-ratios and CVs are identical in studies 1…n. Hence, p(1) = … =p(n). If the outcomes of studies are independent, is p(pass all) = Π p(i), e.g., for p(i=1…6) = 0.90 = 0.906 ~ 0.53?
Or does each study stand on its own and we don’t have to care?

Dif-tor heh smusma 🖖
Helmut Schütz

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
ElMaestro
★★★

Denmark,
2021-02-19 12:57
(75 d 12:32 ago)

@ Helmut
Posting: # 22215
Views: 942

## Probability to pass multiple studies

Hi Helmut,

» say, we have n studies, each powered at 90%. What is the probability (i.e., power) that all of them pass BE?
»
» Let’s keep it simple: T/R-ratios and CVs are identical in studies 1…n. Hence, p(1) = … =p(n). If the outcomes of studies are independent, is p(pass all) = Π p(i), e.g., for p(i=1…6) = 0.90 = 0.906 ~ 0.53?
» Or does each study stand on its own and we don’t have to care?

Yes to 0.53.
The risk is up to you or your client. I think there is no general awareness, but my real worry is the type I error, as I have indicated elsewhere.
"Have to care" really involves the fine print. I think in the absence of further info it is difficult to tell if you should care and/or from which perspective care is necessary.

Related issue, the one that worries me more:
You test one formulation, it fails on the 90% CI, you develop a new formulation, it passes on the 90% CI. What is the type I error? Well, strictly speaking that would be inflated. But noone seems to give a damn.

Pass or fail!
ElMaestro
Helmut
★★★

Vienna, Austria,
2021-02-19 13:37
(75 d 11:51 ago)

@ ElMaestro
Posting: # 22216
Views: 799

## Power limbo

Hi ElMaestro,

» Yes to 0.53.

Shit, expected that.

» The risk is up to you or your client.

Such a case is not uncommon. Say, you have single dose studies (highest strength, fasting/fed), multiple dose studies with two strengths (fasting/fed). Then you get the n = 6 of my example.
If you want a certain overall power, then each study has to be powered to $$p_i=\sqrt[n]{p_\textrm{overall}}$$.
With n = 6, for 80% → 96.35% and for 90% → 98.26%. Will an IEC accept that?*

» I think there is no general awareness, but my real worry is the type I error, as I have indicated elsewhere.

I know.

» Related issue, the one that worries me more:
» You test one formulation, it fails on the 90% CI, you develop a new formulation, it passes on the 90% CI. What is the type I error?

I’m not concerned about a new formulation. The first one went to the waste bin → zero consumer risk. The study supporting the new formulation stands on its own. Hence, TIE ≤0.05.

» Well, strictly speaking that would be inflated. But noone seems to give a damn.

I’m indeed concerned about repeating the study (same formulation) with more subjects. Then you get an inflated TIE.

We have that everywhere. A value in the post-study exam is clinically significant out of range. Follow-up initiated and now all is good. Did the value really improve? There is always inaccuracy involved. Maybe the first one was correct and the second one not.

My doctor gave me six months to live,
but when I couldn’t pay the bill
he gave me six months more.
Walter Matthau

• The IEC assesses protocols of single studies. If we want to keep the desired overall power, sample sizes will be substantially higher:

library(PowerTOST) CV     <- seq(0.2, 0.4, 0.1) ns     <- 1:6L target <- c(0.8, 0.9) theta0 <- 0.95 design <- "2x2x2" res    <- data.frame(CV = rep(CV, each = length(ns)),                      target = rep(target, each = length(CV)*length(ns)),                      n.single = NA, pwr.single = NA, studies = ns) res$target.adj <- res$target^(1/res$studies) for (j in 1:nrow(res)) { res[j, 3:4] <- sampleN.TOST(CV = res$CV[j], theta0 = theta0,                                   design = design,                                   targetpower = res$target[j], print = FALSE)[7:8] tmp <- sampleN.TOST(CV = res$CV[j], theta0 = theta0,                                   design = design,                                   targetpower = res$target.adj[j], print = FALSE) res$n[j]        <- tmp[["Sample size"]]   res$pwr.each[j] <- tmp[["Achieved power"]]^res$studies[j] } res$n.incr <- sprintf("%+.1f%%", 100*(res$n - res$n.single) / res$n) res[, 1:8] <- signif(res[, 1:8], 5) print(res, row.names = FALSE)   CV target n.single pwr.single studies target.adj   n pwr.each n.incr  0.2    0.8       20    0.83468       1    0.80000  20  0.83468  +0.0%  0.2    0.8       20    0.83468       2    0.89443  24  0.80286 +16.7%  0.2    0.8       20    0.83468       3    0.92832  28  0.81705 +28.6%  0.2    0.8       20    0.83468       4    0.94574  30  0.80973 +33.3%  0.2    0.8       20    0.83468       5    0.95635  32  0.81341 +37.5%  0.2    0.8       20    0.83468       6    0.96349  34  0.82384 +41.2%  0.3    0.8       40    0.81585       1    0.80000  40  0.81585  +0.0%  0.3    0.8       40    0.81585       2    0.89443  52  0.81354 +23.1%  0.3    0.8       40    0.81585       3    0.92832  58  0.80091 +31.0%  0.3    0.8       40    0.81585       4    0.94574  64  0.80870 +37.5%  0.3    0.8       40    0.81585       5    0.95635  68  0.80855 +41.2%  0.3    0.8       40    0.81585       6    0.96349  72  0.81548 +44.4%  0.4    0.8       66    0.80525       1    0.80000  66  0.80525  +0.0%  0.4    0.8       66    0.80525       2    0.89443  88  0.81075 +25.0%  0.4    0.8       66    0.80525       3    0.92832 100  0.80737 +34.0%  0.4    0.8       66    0.80525       4    0.94574 108  0.80202 +38.9%  0.4    0.8       66    0.80525       5    0.95635 116  0.80809 +43.1%  0.4    0.8       66    0.80525       6    0.96349 122  0.81015 +45.9%  0.2    0.9       26    0.91763       1    0.90000  26  0.91763  +0.0%  0.2    0.9       26    0.91763       2    0.94868  32  0.92071 +18.8%  0.2    0.9       26    0.91763       3    0.96549  34  0.90766 +23.5%  0.2    0.9       26    0.91763       4    0.97400  36  0.90405 +27.8%  0.2    0.9       26    0.91763       5    0.97915  38  0.90639 +31.6%  0.2    0.9       26    0.91763       6    0.98259  40  0.91230 +35.0%  0.3    0.9       52    0.90197       1    0.90000  52  0.90197  +0.0%  0.3    0.9       52    0.90197       2    0.94868  66  0.90937 +21.2%  0.3    0.9       52    0.90197       3    0.96549  72  0.90304 +27.8%  0.3    0.9       52    0.90197       4    0.97400  78  0.90750 +33.3%  0.3    0.9       52    0.90197       5    0.97915  82  0.90779 +36.6%  0.3    0.9       52    0.90197       6    0.98259  84  0.90158 +38.1%  0.4    0.9       88    0.90041       1    0.90000  88  0.90041  +0.0%  0.4    0.9       88    0.90041       2    0.94868 110  0.90173 +20.0%  0.4    0.9       88    0.90041       3    0.96549 122  0.90008 +27.9%  0.4    0.9       88    0.90041       4    0.97400 132  0.90364 +33.3%  0.4    0.9       88    0.90041       5    0.97915 138  0.90121 +36.2%  0.4    0.9       88    0.90041       6    0.98259 144  0.90267 +38.9% n.single   Number of subjects for given CV and target power in a single study
pwr.single Achieved power with n.single subjects in a single study
target.adj Adjusted target power for sample size estimation of multiple studies
n          Number of subjects for given CV and adjusted power in each of the studies
pwr.each   Overall power of the studies
n.incr     Penalty in sample size compared to a single study

Example:
We assume CV 20%, T/R-ratio 0.95, and desire a power of ≥80%. Then in a single study with 20 subjects we would achieve a power of ~83.5%. If we want to perform six studies, the overall power would drop to $$\small{0.80^6\approx 26.2\%}$$. In order to pass all studies with ≥80% power, we have to adjust the target power to $$\small{\sqrt[6]{0.80}=0.96349}$$. Now we need 34 subjects in each of the studies.

Dif-tor heh smusma 🖖
Helmut Schütz

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes