## The secret of WinNonlin's BE's post-hoc Power analysis [Software]

Dear all,

As asked by HS and provoked by this, it's time to talk about WNL's BE's post-hoc Power analysis.

For a long time, WNL's Power is considered as statistical NONSENSE, see:
HS's presentations such as

Of course, I agree with our HS's opinion on this issue largely because of the strange method used by WNL for Power calculation, and the strange results as well.

Actually, I have consulted a medical statistician who is professional at design and analysis of clinical trials. He told me he usually do power analysis to determine sample size before starting a trial, but seldom calc post-hoc power, he also told me that even if the post-hoc power is a little lower than designed or expected power, it is no necessary to do additional trial to increase the sample size and expect to increase the power.
Consistent with HS’s and Potvin's opinions.

Regardless of the necessity, let's first look inside into WNL to see how it calc power.

----------From PNX WNL User Guide P341-342----------------------------------
Power
Power is the post-hoc probability of detecting a difference greater than or equal to a specified percentage of the reference least squares mean. In general,

Power = 1 – (probability of a type II error)
= probability of rejecting H0 when H1 is true.

In the bioequivalence calculations, the hypotheses being tested are:

H0: RefLSM = TestLSM
H1: RefLSM ≠ TestLSM

For the no-transform case, the power is the probability of rejecting H0 given:

|TestLSM – RefLSM | ≥ fractionToDetect × RefLSM

For ln-transform, and data already ln-transformed, this changes to:

|TestLSM – RefLSM | ≥ – ln(1 – fractionToDetect),

and similarly for log10-transform and data already log10-transformed.

For the sake of illustration, assume fractionToDetect = 0.2, RefLSM>0, and no transform was done on the data. Also the maximum probability of rejecting H0 occurs in the case of equality,
|TestLSM – RefLSM| = 0.2×RefLSM. So:

Power = Pr (rejecting H0 at the alpha level given |Difference| = 0.2×RefLSM)
= Pr (|Difference/DiffSE| > t(1-α/2),df given |Difference| = 0.2×RefLSM)
= Pr (Difference/DiffSE > t(1-α/2),df given |Difference| = 0.2×RefLSM)
+ Pr (Difference/DiffSE < –t(1-α/2),df given |Difference| = 0.2×RefLSM)

Let:

t1 = t(1-α/2),df – 0.2×RefLSM / DiffSE
t2 = – t(1-α/2),df – 0.2×RefLSM / DiffSE
tstat = (Difference – 0.2×RefLSM) / DiffSE

Then:

Power = Pr(tstat > t1 given |Difference| = 0.2×RefLSM)
+ Pr(tstat < t2 given |Difference| = 0.2×RefLSM)

Note that tstat is T-distributed. Let p1 be the p-value associated with t1 (the area in the tail), and let p2 be the p-value associated with t2 (the area in the tail; note that p2 may be negligible). Then:

Power = 1 – (p1 – p2)

----------From PNX WNL User Guide P341-342----------------------------------
regretfully, WNL doesn't give us the formula when data is ln=transformed before BE analysis.

Before we start to validate WNL's formula, let me show my first puzzle:
As pointed by above WNL's user guide, it is obviouse that WNL is calculating the Power for TOST, but in WNL's ASCII output WNL gives us
 Power of ANOVA for Confidence Level 90.00     Power at 20% = 0.993541 
for ANOVA or for TOST? It is a question.

Now, let me validate WNL's formula using Chow and Liu's dataset.

-----For original data without transformation--------
WNL's result is
 Power of ANOVA for Confidence Level 90.00        Power at 20% = 0.9935408 

My formula and result is:
t1 = t (1-α/2),df – (1 - 0.8)×RefLSM / DiffSE t2 = – t (1-α/2),df – (1.2 -1)×RefLSM / DiffSE p1 = TDist(Abs(t1), n1 + n2 - 2, 1) p2 = TDist(Abs(t2), n1 + n2 - 2, 1) Power = 1 - (p1 - p2)

Power Analysis -------------------------------------------------------------------------- Parameter  T stat 1     T stat 2     P value 1     P value 2     Power -------------------------------------------------------------------------- AUC      -2.705765713 -6.140054462  0.006455207   0.000001760  0.993546553 --------------------------------------------------------------------------

------------For ln-transformed data---------------
WNL's result is
  Power of ANOVA for Confidence Level 90.00         Power at 20% = 0.9839865

My formula and result is:
t1 = t (1-α/2),df + Ln(0.8) / DiffSE t2 = – t (1-α/2),df - Ln(1.25) / DiffSE p1 = TDist(Abs(t1), n1 + n2 - 2, 1) p2 = TDist(Abs(t2), n1 + n2 - 2, 1) Power = 1 - (p1 - p2)

Power Analysis -------------------------------------------------------------------------- Parameter  T stat 1      T stat 2     P value 1     P value 2     Power -------------------------------------------------------------------------- ln(AUC)  -2.289520539  -5.723809288  0.016004731   4.65843E-06   0.983999927 --------------------------------------------------------------------------

To ensure the sufficient precision of my calc, all the above calculations were done using MS Excel 2010 which is of much more higher precision than MS Excel 2000, 2003 and 2007 when Tinv() or Tdist() is used.
For comparison:
   Software                Tinv(0.1,22) R 2.10.1                1.717144374380243 Excel 2010              1.71714437438024 Open Office 3.3.0       1.71714437438025 Gnumeric 1.10.16        1.71714437438148 Excel 2003 and 2007     1.71714433543983 WNL 5.1.1               1.71714434835526

It should be noted that the formula I used for Ln-transformed data is of no reference, only obtained by my guess, but it works well specially to obtain WNL's results. I don't know whether it is correct or not and don't know WNL is correct or not.

If we using PowerTOST as a gold standard for power calc of TOST, all WNL's results are and totally wrong.

It seems that WNL's post-hoc power calc doesn't directly use Observed Diff or Observed Ratio or intra-CV, only indirectly uses DiffSE and Expected Diff or Ratio such as 0.8, 1,2, 0.8, 1.25. Dear all, True or False of WNL's method?

Dear HS, have you reached Berlin and meet our D. Labes? I need your insight and comments on this issue.