The secret of WinNonlin's BE's post-hoc Power analysis [Software]

posted by yicaoting  – NanKing, China, 2011-11-08 15:40 (4523 d 20:33 ago) – Posting: # 7636
Views: 10,837

Dear all,

As asked by HS and provoked by this, it's time to talk about WNL's BE's post-hoc Power analysis.

For a long time, WNL's Power is considered as statistical NONSENSE, see:
HS's presentations such as
this page 26,
this page 12, and
this page 22.

Of course, I agree with our HS's opinion on this issue largely because of the strange method used by WNL for Power calculation, and the strange results as well.

Actually, I have consulted a medical statistician who is professional at design and analysis of clinical trials. He told me he usually do power analysis to determine sample size before starting a trial, but seldom calc post-hoc power, he also told me that even if the post-hoc power is a little lower than designed or expected power, it is no necessary to do additional trial to increase the sample size and expect to increase the power.
Consistent with HS’s and Potvin's opinions.

Regardless of the necessity, let's first look inside into WNL to see how it calc power.

----------From PNX WNL User Guide P341-342----------------------------------
Power
Power is the post-hoc probability of detecting a difference greater than or equal to a specified percentage of the reference least squares mean. In general,

Power = 1 – (probability of a type II error)
= probability of rejecting H0 when H1 is true.


In the bioequivalence calculations, the hypotheses being tested are:

H0: RefLSM = TestLSM
H1: RefLSM ≠ TestLSM


For the no-transform case, the power is the probability of rejecting H0 given:

|TestLSM – RefLSM | ≥ fractionToDetect × RefLSM


For ln-transform, and data already ln-transformed, this changes to:

|TestLSM – RefLSM | ≥ – ln(1 – fractionToDetect),


and similarly for log10-transform and data already log10-transformed.

For the sake of illustration, assume fractionToDetect = 0.2, RefLSM>0, and no transform was done on the data. Also the maximum probability of rejecting H0 occurs in the case of equality,
|TestLSM – RefLSM| = 0.2×RefLSM. So:

Power = Pr (rejecting H0 at the alpha level given |Difference| = 0.2×RefLSM)
= Pr (|Difference/DiffSE| > t(1-α/2),df given |Difference| = 0.2×RefLSM)
= Pr (Difference/DiffSE > t(1-α/2),df given |Difference| = 0.2×RefLSM)
+ Pr (Difference/DiffSE < –t(1-α/2),df given |Difference| = 0.2×RefLSM)


Let:

t1 = t(1-α/2),df – 0.2×RefLSM / DiffSE
t2 = – t(1-α/2),df – 0.2×RefLSM / DiffSE
tstat = (Difference – 0.2×RefLSM) / DiffSE


Then:

Power = Pr(tstat > t1 given |Difference| = 0.2×RefLSM)
+ Pr(tstat < t2 given |Difference| = 0.2×RefLSM)


Note that tstat is T-distributed. Let p1 be the p-value associated with t1 (the area in the tail), and let p2 be the p-value associated with t2 (the area in the tail; note that p2 may be negligible). Then:

Power = 1 – (p1 – p2)


----------From PNX WNL User Guide P341-342----------------------------------
regretfully, WNL doesn't give us the formula when data is ln=transformed before BE analysis.

Before we start to validate WNL's formula, let me show my first puzzle:
As pointed by above WNL's user guide, it is obviouse that WNL is calculating the Power for TOST, but in WNL's ASCII output WNL gives us
Power of ANOVA for Confidence Level 90.00
    Power at 20% = 0.993541

for ANOVA or for TOST? It is a question.

Now, let me validate WNL's formula using Chow and Liu's dataset.

-----For original data without transformation--------
WNL's result is
Power of ANOVA for Confidence Level 90.00
       Power at 20% = 0.9935408


My formula and result is:
t1 = t (1-α/2),df – (1 - 0.8)×RefLSM / DiffSE
t2 = – t (1-α/2),df – (1.2 -1)×RefLSM / DiffSE
p1 = TDist(Abs(t1), n1 + n2 - 2, 1)
p2 = TDist(Abs(t2), n1 + n2 - 2, 1)
Power = 1 - (p1 - p2)


Power Analysis
--------------------------------------------------------------------------
Parameter  T stat 1     T stat 2     P value 1     P value 2     Power
--------------------------------------------------------------------------
AUC      -2.705765713 -6.140054462  0.006455207   0.000001760  0.993546553
--------------------------------------------------------------------------



------------For ln-transformed data---------------
WNL's result is
  Power of ANOVA for Confidence Level 90.00
        Power at 20% = 0.9839865


My formula and result is:
t1 = t (1-α/2),df + Ln(0.8) / DiffSE
t2 = – t (1-α/2),df - Ln(1.25) / DiffSE
p1 = TDist(Abs(t1), n1 + n2 - 2, 1)
p2 = TDist(Abs(t2), n1 + n2 - 2, 1)
Power = 1 - (p1 - p2)


Power Analysis
--------------------------------------------------------------------------
Parameter  T stat 1      T stat 2     P value 1     P value 2     Power
--------------------------------------------------------------------------
ln(AUC)  -2.289520539  -5.723809288  0.016004731   4.65843E-06   0.983999927
--------------------------------------------------------------------------


To ensure the sufficient precision of my calc, all the above calculations were done using MS Excel 2010 which is of much more higher precision than MS Excel 2000, 2003 and 2007 when Tinv() or Tdist() is used.
For comparison:
   Software                Tinv(0.1,22)
R 2.10.1                1.717144374380243
Excel 2010              1.71714437438024
Open Office 3.3.0       1.71714437438025
Gnumeric 1.10.16        1.71714437438148
Excel 2003 and 2007     1.71714433543983
WNL 5.1.1               1.71714434835526


It should be noted that the formula I used for Ln-transformed data is of no reference, only obtained by my guess, but it works well specially to obtain WNL's results. I don't know whether it is correct or not and don't know WNL is correct or not.

If we using PowerTOST as a gold standard for power calc of TOST, all WNL's results are :confused: and totally wrong.

It seems that WNL's post-hoc power calc doesn't directly use Observed Diff or Observed Ratio or intra-CV, only indirectly uses DiffSE and Expected Diff or Ratio such as 0.8, 1,2, 0.8, 1.25. Dear all, True or False of WNL's method?

Dear HS, have you reached Berlin and meet our D. Labes? I need your insight and comments on this issue.

Complete thread:

UA Flag
Activity
 Admin contact
22,957 posts in 4,819 threads, 1,639 registered users;
90 visitors (0 registered, 90 guests [including 7 identified bots]).
Forum time: 12:13 CET (Europe/Vienna)

Nothing shows a lack of mathematical education more
than an overly precise calculation.    Carl Friedrich Gauß

The Bioequivalence and Bioavailability Forum is hosted by
BEBAC Ing. Helmut Schütz
HTML5