Bioequivalence and Bioavailability Forum

Dear all,

As asked by HS and provoked by this, it's time to talk about WNL's BE's post-hoc Power analysis.

For a long time, WNL's Power is considered as statistical NONSENSE, see:
HS's presentations such as
this page 26,
this page 12, and
this page 22.

Of course, I agree with our HS's opinion on this issue largely because of the strange method used by WNL for Power calculation, and the strange results as well.

Actually, I have consulted a medical statistician who is professional at design and analysis of clinical trials. He told me he usually do power analysis to determine sample size before starting a trial, but seldom calc post-hoc power, he also told me that even if the post-hoc power is a little lower than designed or expected power, it is no necessary to do additional trial to increase the sample size and expect to increase the power.
Consistent with HS’s and Potvin's opinions.

Regardless of the necessity, let's first look inside into WNL to see how it calc power.

----------From PNX WNL User Guide P341-342----------------------------------
Power
Power is the post-hoc probability of detecting a difference greater than or equal to a specified percentage of the reference least squares mean. In general,

Power = 1 – (probability of a type II error)
= probability of rejecting H₀ when H₁ is true.

In the bioequivalence calculations, the hypotheses being tested are:

H₀: RefLSM = TestLSM
H₁: RefLSM ≠ TestLSM

For the no-transform case, the power is the probability of rejecting H₀ given:

|TestLSM – RefLSM | ≥ fractionToDetect × RefLSM

For ln-transform, and data already ln-transformed, this changes to:

|TestLSM – RefLSM | ≥ – ln(1 – fractionToDetect),

and similarly for log10-transform and data already log10-transformed.

For the sake of illustration, assume fractionToDetect = 0.2, RefLSM>0, and no transform was done on the data. Also the maximum probability of rejecting H₀ occurs in the case of equality,
|TestLSM – RefLSM| = 0.2×RefLSM. So:

Power = Pr (rejecting H₀ at the alpha level given |Difference| = 0.2×RefLSM)
= Pr (|Difference/DiffSE| > t_(1-α/2),df given |Difference| = 0.2×RefLSM)
= Pr (Difference/DiffSE > t_(1-α/2),df given |Difference| = 0.2×RefLSM)
+ Pr (Difference/DiffSE < –t_(1-α/2),df given |Difference| = 0.2×RefLSM)

Let:

t₁ = t_(1-α/2),df – 0.2×RefLSM / DiffSE
t₂ = – t_(1-α/2),df – 0.2×RefLSM / DiffSE
t_stat = (Difference – 0.2×RefLSM) / DiffSE

Then:

Power = Pr(t_stat > t₁ given |Difference| = 0.2×RefLSM)
+ Pr(t_stat < t₂ given |Difference| = 0.2×RefLSM)

Note that t_stat is T-distributed. Let p₁ be the p-value associated with t₁ (the area in the tail), and let p₂ be the p-value associated with t₂ (the area in the tail; note that p₂ may be negligible). Then:

Power = 1 – (p₁ – p₂)

----------From PNX WNL User Guide P341-342----------------------------------
regretfully, WNL doesn't give us the formula when data is ln=transformed before BE analysis.

Before we start to validate WNL's formula, let me show my first puzzle:
As pointed by above WNL's user guide, it is obviouse that WNL is calculating the Power for TOST, but in WNL's ASCII output WNL gives us
Power of ANOVA for Confidence Level 90.00
    Power at 20% = 0.993541
for ANOVA or for TOST? It is a question.

Now, let me validate WNL's formula using Chow and Liu's dataset.

-----For original data without transformation--------
WNL's result is
Power of ANOVA for Confidence Level 90.00
       Power at 20% = 0.9935408

My formula and result is:
t1 = t (1-α/2),df – (1 - 0.8)×RefLSM / DiffSE
t2 = – t (1-α/2),df – (1.2 -1)×RefLSM / DiffSE
p1 = TDist(Abs(t1), n1 + n2 - 2, 1)
p2 = TDist(Abs(t2), n1 + n2 - 2, 1)
Power = 1 - (p1 - p2)

Power Analysis
--------------------------------------------------------------------------
Parameter  T stat 1     T stat 2     P value 1     P value 2     Power
--------------------------------------------------------------------------
AUC      -2.705765713 -6.140054462  0.006455207   0.000001760  0.993546553
--------------------------------------------------------------------------


------------For ln-transformed data---------------
WNL's result is
  Power of ANOVA for Confidence Level 90.00
        Power at 20% = 0.9839865

My formula and result is:
t1 = t (1-α/2),df + Ln(0.8) / DiffSE
t2 = – t (1-α/2),df - Ln(1.25) / DiffSE
p1 = TDist(Abs(t1), n1 + n2 - 2, 1)
p2 = TDist(Abs(t2), n1 + n2 - 2, 1)
Power = 1 - (p1 - p2)

Power Analysis
--------------------------------------------------------------------------
Parameter  T stat 1      T stat 2     P value 1     P value 2     Power
--------------------------------------------------------------------------
ln(AUC)  -2.289520539  -5.723809288  0.016004731   4.65843E-06   0.983999927
--------------------------------------------------------------------------

To ensure the sufficient precision of my calc, all the above calculations were done using MS Excel 2010 which is of much more higher precision than MS Excel 2000, 2003 and 2007 when Tinv() or Tdist() is used.
For comparison:
   Software                Tinv(0.1,22)
R 2.10.1                1.717144374380243
Excel 2010              1.71714437438024
Open Office 3.3.0       1.71714437438025
Gnumeric 1.10.16        1.71714437438148
Excel 2003 and 2007     1.71714433543983
WNL 5.1.1               1.71714434835526

It should be noted that the formula I used for Ln-transformed data is of no reference, only obtained by my guess, but it works well specially to obtain WNL's results. I don't know whether it is correct or not and don't know WNL is correct or not.

If we using PowerTOST as a gold standard for power calc of TOST, all WNL's results are and totally wrong.

It seems that WNL's post-hoc power calc doesn't directly use Observed Diff or Observed Ratio or intra-CV, only indirectly uses DiffSE and Expected Diff or Ratio such as 0.8, 1,2, 0.8, 1.25. Dear all, True or False of WNL's method?

Dear HS, have you reached Berlin and meet our D. Labes? I need your insight and comments on this issue.