Bioequivalence and Bioavailability Forum

Hi ElMaestro,

❝ Could you tell how you got your F-test denominator? I am sure you are right, but I don't know where it came from. For an interaction of a between- and within-factor I think the rule of thumb (which is also a wee bit hard to define ) is to test against the within, which in this case would be the model residual.

Yep. Below an example of model 1 in Phoenix/WinNonlin. Two groups (n=24 each), all effects fixed.

Partial Sum of Squares
            Hypothesis DF        SS        MS   F_stat  P_value
---------------------------------------------------------------
                 Group  1 0.0131109 0.0131109 1.0385149 0.31374
              Sequence  1 0.0058638 0.0058638 0.4644731 0.49914
             Treatment  1 0.0011965 0.0011965 0.0947752 0.75964
        Group*Sequence  1 0.0108734 0.0108734 0.8612869 0.35844
          Group*Period  2 0.0160976 0.0080488 0.6375490 0.53340
       Group*Treatment  1 0.0131109 0.0131109 1.0385149 0.31374
Group*Sequence*Subject 44 0.555484  0.0126246 1         0.50000
                 Error 44 0.555484  0.0126246

Partial Tests of Model Effects
            Hypothesis Numer_DF Denom_DF  F_stat   P_value
----------------------------------------------------------
                 Group        1       44 1.0385149 0.31374
              Sequence        1       44 0.4644731 0.49914
             Treatment        1       44 0.0947752 0.75964
        Group*Sequence        1       44 0.8612869 0.35844
          Group*Period        2       44 0.6375490 0.53340
       Group*Treatment        1       44 1.0385149 0.31374
Group*Sequence*Subject       44       44 1         0.50000

N: Σn_G = 48
G: 2
Numerator DF: G – 1 = 1
Denominator DF: N - 2G = 44
F: 0.0131109/0.0126246 = 1.0385149
round(pf(1.0385149, 1, 44, lower.tail=FALSE), 5)
# [1] 0.31374 ✔

Hi mittyri,

❝ ❝ The idea behind the Group-by-Treatment interaction is that the T/R in one group is different from the other (i.e., we have collinearity with a “hidden” variable). Therefore, simulate a group of subjects with T/R 0.95 and another one with T/R 0.95^–1 (CV ad libitum). Merge them to get a “study”. Run model 1 and check the p-value of the Group-by-Treatment interaction. With the simple model you should expect T/R 1.

❝

❝ seems to be reasonable, but I do not see why the power is low?

Good question. Next question?

I performed simulations (100,000 2×2×2 studies each for conditions a. and b. specified below). Two groups of 16 subjects each, CV 30%, no period and sequence effects. 32 subjects should give power 81.52% for T/R 1. If the Group-by-Treatment interaction is not significant (p ≥0.1) in model 1, the respective study is evaluated by model 2 (pooled data) or both groups by model 3 otherwise. All studies are evaluated by model 3 (pooled data). The listed PE is the geometric mean of passing studies’ PEs.

1. T/R in group 1 0.95, T/R in group 2 0.95^–1
   (i.e., ‘true’ Group-by-Treatment interaction):
   Model 1: p(G×T) <0.1 in 17.91% of studies.
Evaluation of studies with p(G×T) <0.1 (Groups):
  passed model 3 (1)      :  1.42% (of tested); PE  98.69%
                                  range of PEs: 92.45% to 107.63%
  passed model 3 (2)      :  1.64% (of tested); PE 100.99%
                                  range of PEs: 93.99% to 108.23%
  passed model 3 (1 and 2):  0.00% (of tested)
Evaluation of studies with p(G×T) ≥0.1 (pooled):
  passed model 2          : 66.47% (overall)
                            80.97% (of tested); PE  99.97%
                                  range of PEs: 86.36% to 114.27%
Studies passing any of model 2 or 3: 67.02%
Criteria for simple model fulfilled:
  passed model 3          : 80.95%;             PE  99.98%
                                  range of PEs: 86.36% to 114.68%
   
   
2. T/R in both groups 1.00
   (i.e., no Group-by-Treatment interaction):
   Model 1: p(G×T) <0.1 in 9.79% of studies.
Evaluation of studies with p(G×T) <0.1 (Groups):
  passed model 3 (1)      :  1.86% (of tested); PE 100.28%
                                  range of PEs: 93.09% to 108.40%
  passed model 3 (2)      :  1.87% (of tested); PE 100.01%
                                  range of PEs: 92.18% to 108.41%
  passed model 3 (1 and 2):  0.00% (of tested)
Evaluation of studies with p(G×T) ≥0.1 (pooled):
  passed model 2          : 73.33% (overall)
                            81.28% (of tested); PE  99.98%
                                  range of PEs: 86.36% to 114.68%
Studies passing any of model 2 or 3: 73.69%
Criteria for simple model fulfilled:
  passed model 3          : 81.40%;             PE  99.98%
                                  range of PEs: 86.36% to 115.15%
   

IMHO, equal groups sizes are problematic. What if one group passes and the other fails? Even if one is fishy and present only the passing one, assessors likely would ask for the other one and make a conservative decision. Hoping that both groups will pass is simply futile.

Lessons learned:
If we test at the 10% level and there is no true Group-by-Treatment interaction we will find a significant effect at ~ the level of the test – as expected (b). Hurray, false positives!
On the other hand, if there is one, we will detect it (a).
The percentage of studies passing in models 2 and 3 are similar. Theoretically in model 2 it should be slightly lower than in model 3 (one degree of freedom of the treatment effect less). However, overall power is seriously compromised.

Slowly I get the impression that the evaluation of groups (by model 3) is not a good idea. If there is a true Group-by-Treatment interaction why the heck should the PE (say in the largest group) be unbiased? I would rather say that if one believes that a Group-by-Treatment interaction really exists (I don’t) and the test makes sense (I don’t) evaluation (of the largest group) by model 3 should not be performed. Consequently ~⅒ of (otherwise passing) studies would go into the waste bin. Didn’t I say that before?

---

The distribution of p-values should be uniform.
Looks good for b.



     Min.   1st Qu.    Median      Mean   3rd Qu.      Max.
0.0000011 0.2517777 0.5002957 0.5008763 0.7508297 0.9999974

Interesting shape for a.



     Min.   1st Qu.    Median      Mean   3rd Qu.      Max.
0.0000001 0.1562932 0.3991516 0.4306846 0.6868190 0.9999981

If you prefer more extreme stuff: T/R in group 1 0.90, T/R in group 2 0.90^–1

Model 1: p(G×T) <0.1 in 40.35% of studies.
Evaluation of studies with p(G×T) <0.1 (Groups):
  passed model 3 (1)      :  1.09% (of tested); PE  98.76%
                                  range of PEs: 91.69% to 105.97%
  passed model 3 (2)      :  1.06% (of tested); PE 101.40%
                                  range of PEs: 94.58% to 108.34%
  passed model 3 (1 and 2):  0.00% (of tested)
Evaluation of studies with p(G×T) ≥0.1 (pooled):
  passed model 2          : 47.74% (overall)
                            80.03% (of tested); PE  99.98%
                                  range of PEs: 87.24% to 114.13%
Studies passing any of model 2 or 3: 48.60%
Criteria for simple model fulfilled:
  passed model 3          : 79.45%;             PE  99.99%
                                  range of PEs: 87.24% to 114.13%

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
0.00000 0.03962 0.15648 0.26602 0.42742 0.99997

---

PS: The code seems to work – at least for the pooled model 3. Comparisons of powers
power.TOST(...)                0.815152
power.TOST.sim(..., nsims=1e5) 0.81437
power.TOST.sim(..., nsims=1e6) 0.815127
My code (nsims=1e5)            0.81402
My code (nsims=1e6)            0.81551

Hi mittyri and all,

R-code to estimate the loss in power (two groups of equal size). Example for my simulations above and assuming that we will get a significant Group-by-Treatment interaction at the level of the test.

library(PowerTOST)
CV          <- 0.3
theta0      <- 1
targetpower <- 0.8
p.level     <- 0.1
res         <- sampleN.TOST(CV=CV, theta0=theta0, targetpower=targetpower,
                            print=FALSE)
N           <- res[["Sample size"]]
if (N >= 12) { # at least minimum sample size acc. to GLs?
  pwr.mod3pooled <- res[["Achieved power"]]
} else {
  N <- 12
  pwr.mod3pooled <- power.TOST(CV=CV, theta0=theta0, n=N)
}
pwr.mod2       <- suppressMessages(power.TOST(CV=CV, theta0=theta0,
                                              n=N-1)*(1-p.level))
pwr.mod3groups <- power.TOST(CV=CV, theta0=theta0, n=N/2)*p.level
pwr.mod2and3   <- pwr.mod2+pwr.mod3groups
cat(sprintf("CV %5.3f%%, theta0 %.4f, targetpower %.2f%%    : sample size %i",
            100*CV, theta0, 100*targetpower, N),
    "\nLevel of the G\u00D7T test (model 1)                  :",
      sprintf("% 6.4f", p.level),
     "\nPower of studies evaluated by model 2 (pooled)   :",
      sprintf("%5.2f%%", 100*pwr.mod2),
    "\nPower of studies evaluated by model 3 (groups)   :",
      sprintf("%5.2f%%", 100*pwr.mod3groups),
    "\nModel 2 (pooled) and 3 (groups) combined         :",
      sprintf("%5.2f%%", 100*pwr.mod2and3),
    "\nPower of studies evaluated by model 3 (pooled)   :",
      sprintf("%5.2f%%", 100*pwr.mod3pooled),
    "\nLoss in power if simple model 3 cannot be applied:",
      sprintf("%5.2f%%", 100*(pwr.mod3pooled-pwr.mod2and3)), "\n")

Gives

CV 30.000%, theta0 1.0000, targetpower 80.00%    : sample size 32
Level of the G×T test (model 1)                  :  0.1000
Power of studies evaluated by model 2 (pooled)   : 71.80%
Power of studies evaluated by model 3 (groups)   :  3.25%
Model 2 (pooled) and 3 (groups) combined         : 75.05%
Power of studies evaluated by model 3 (pooled)   : 81.52%
Loss in power if simple model 3 cannot be applied:  6.47%

Hi mittyri and all,

the p-value of the Group-by-Treatment interaction seemingly does not depend on the interval between groups. In 51% of the studies the interval was three days or less but in some substantially longer (i.e., steady state studies where the clinic was occupied). The bubbles’ area, the linear regression, and the loess curve are scaled/weighed by the sample size.

slope: 0.011196 (p 0.165)



slope: 0.010589 (p 0.186)

Therefore, we should not worry. The FDA defines “greatly separated in time” as “months apart, for example”.

Hi mittyri,

continuing the evaluation of the data sets of this post.
Background: Some studies are quite dated (the oldest performed in October 1992!). In those days a pre-specified acceptance range of 75.00–133.33% (or even 70.00–142.86%) was acceptable for C_max. However, I evaluated all data sets for the common 80.00–125.00%. This explains why more than the expected 20% failed (no, I didn’t screw up the design ). If ever possible I tried to avoid equal group sizes but kept one as large as possible. Didn’t succeed sometimes. Working in a CRO, the sponsor is always right…
All data sets were evaluated by model 3 for pooled data (like in the reports – I never cared about groups) and by model 1 to get the p value of the Group-by-Treatment interaction.

• If p (G×T) ≥0.1 the pooled data was evaluated by model 2.
  
• If p (G×T) <0.1 the largest group(s) were evaluated by model 3.
  If there were more than one large group with equal sizes, both had to pass since I expect assessors will ask for it.

Here the results:

86 studies, 60 analytes, data sets: 85 (AUC), 86 (Cmax).
Evaluated by model 1 (all effects fixed); p (G×T) <0.1:
AUC :   8.24% ( 7/85)
Cmax:  12.79% (11/86)

Summary of passing results.

AUC : model 2 (pooled)                           :  84.62% (66/78)
      model 2 (pooled without pre-test)          :  84.71% (72/85)
              loss (compared to pooled model 3)  :   1.18% ( 1/85)
      model 3 (largest group)                    :  85.71% ( 6/ 7)
      model 2 (pooled) or model 3 (largest group):  84.71% (72/85)
              loss (compared to pooled model 3)  :   1.18% ( 1/85)
      model 3 (pooled)                           :  85.88% (73/85)
              CV (range)                         :  21.36% (4.59–61.73%)

Cmax: model 2 (pooled)                           :  62.67% (47/75)
      model 2 (pooled without pre-test)          :  63.95% (55/86)
              loss (compared to pooled model 3)  :   0.00% ( 0/86)
      model 3 (largest group)                    :  27.27% ( 3/11)
      model 2 (pooled) or model 3 (largest group):  58.14% (50/86)
              loss (compared to pooled model 3)  :   5.81% ( 5/86)
      model 3 (pooled)                           :  63.95% (55/86)
              CV (range)                         :  27.88% (6.82–76.99%)

The loss in power if we follow the FDA’s procedure (compared to the pooled model 3) is lower than I expected. Surprise. A possible explanation is that studies were usually powered for C_max. Therefore, already the largest groups passed AUC.
On another note: If we apply model 2 without a pre-test (maybe the best way to go for regulators insisting in a group-term) the loss in power compared to the pooled model 3 is negligible. Reasonable, since we lost only few residual degrees of freedom:
pooled model 3: DF=n₁+n₂–2
pooled model 2: DF=n₁+n₂-(N_groups–1)–2

Dear Helmut! ()

The problem will remain untill the Eurasian Economic Union requirements will be corrected or modified. And instead of forgetting this subject as a nightmare we see the inverse tendention: new economic association involves new countries into this problem. Now Belarus was dragged into this story... What's next? It might be stopped just after a scientific paper by the respected author will announce the uselessness of this stuff...