in this thread
we have discussed if an adjustment of alpha has to be done if more then two formulations are to be compared.
Result: If the hypotheses to be tested are combined via 'or' there has to be an alpha adjustment applied to avoid an inflation of the overall alpha.
Bonferroni is one possibility, but may be too conservative. During thinking about that came a paper into my mind
Cheng Zheng, JixianWang and Lihui Zhaoa
Testing bioequivalence for multiple formulations with power and sample size calculations
Pharm Stat 2012 Jul-Aug;11(4):334-41. doi:doi: 10.1002/pst.1522
. Epub 2012 Jun 12.
The authors of that paper have developed an R package MATOST wich deals with the implementation of combining multiple comparison adjustment approaches, such as Hochberg's or Dunnett's method, with TOST.
Meanwhile that package is no longer available, but only some spare R code can be found at https://sites.google.com/site/matostbe/
, restricted to the Holm method for p-value adjustment.
I have taken that code and modified it to the aim of incorporating other methods, streamlining the input arguments and tried also to optimize the run-time. Here is the result of my attempt:
library(mvtnorm) # for multivariate normal
# function implementing section 2.3.1 of the MATOST paper
power_MATOST_S = function(alpha=0.05, n, r, CV, adjust=p.adjust.methods, nsims=50000)
# Calculation power based on pvalue adjustment and simulation for multiple contrasts
# The larger p value is selected before applying the adjustment
# n: total sample size (NOT per sequence as in nQuery)
# only for balanced 2x2 design (!)
# r: expected ratios
# CV: intra-subject coefficient of variation (all-at-once ANOVA)
# Sigma: based on (log(CV^2+1)) with proper correlations around 0.5
p <- length(r)
Cor <- diag(rep(1,p))
for (i in 1:p) for (j in 1:p)
if (i!=j) Cor[i,j]=0.5
# attention! first sigma lower case, second upper case S
sigma = rep(sqrt(log(CV^2+1)), p)
Sigma = diag(sigma) %*% Cor %*% diag(sigma)
# reserve memory
flag <- matrix(0, nsims, p)
df_t <- (p+1)*n-(p+1)-(n-1)-p
cfact <- sqrt(2/n) # balanced designs
cfact2 <- 2/n
# acceptance range kog-transformed
lL <- log(0.8)
lU <- log(1.25)
# simulate contrasts
r_me_n <- mvtnorm::rmvnorm(n=nsims, mean=log(r), sigma=Sigma * cfact2)
# simulate mse (all-at-once). only for equal variabilities ?!
s <- sqrt(rchisq(n=nsims, df=df_t) / df_t * Sigma[1,1])
for (i in 1:nsims)
#calculate the two TOST p values for each contrast
p1 <- 1 - sapply((r_me_n[i,] - lL) / (s[i] * cfact), function(x) pt(x, df=df_t))
p2 <- sapply((r_me_n[i,] - lU) / (s[i] * cfact), function(x) pt(x, df=df_t))
# choose maximum as result
p3 <- pmax(p1, p2)
# adjust the p values for multiplicity
p3 <- p.adjust(p3, adjust)
flag[i, ] <- as.numeric(p3 <= alpha)
# what they next do in their code is beyond my understanding! what's alpha?
# return(list(power=apply(flag, 2, sum)/nsims, alpha=sum(apply(flag, 1, sum)>0)/nsims))
# seems they count column-wise (2) the BE decision and call that power
# alpha: sum row-wise (1) and decide if >0 (i.e. 'or'), then sum that
# this would be overall power/alpha for me
pwr <- sum(rowSums(flag)>0)/nsims
pwrs <- colSums(flag)/nsims
names(pwrs) <- paste("T", 1:p, "-R", sep="")
} # end function
Take that code with a grain of salt.
- It is a quickshot without a full understanding what's happen behind the scenes.
- It comes without any checks of the input arguments.
- It is not the fastest code, therefore the default no of simulations is set to 50 000. Too low for an exploration if the TIE.
- Find more limitations in the comments of the code.
Example call for comparing two test formulations with a reference, without multiplicity adjustment, assuming both test formulations are not bioequivalent to the reference:
power_MATOST_S(n=24, r=c(1.25, 1.25), CV=0.25, adjust="none")
(power is the overall probability of deciding BE for the hypothesis BE of T1-R or T2-R,
pwrs the probability of deciding BE for T1 and T2 alone)
As you see, there is an alpha inflation of the overall hypothesis.
Applying a multiplicity adjustment according to the Holm procedure gives:
power_MATOST_S(n=24, r=c(1.25, 1.25), CV=0.25, adjust="holm")
Overall alpha is controlled.
The Bonferroni adjustment gives similar results:
power_MATOST_S(n=24, r=c(1.25, 1.25), CV=0.25, adjust="bonfer")
Happy playing around with this tool
BTW: Don't ask me about the details of the multiplicity adjustment methods. Take a good textbook about that.