yicaoting Regular NanKing, China, 20111108 13:40 (edited by yicaoting on 20111108 14:21) Posting: # 7636 Views: 7,356 

Dear all, As asked by HS and provoked by this, it's time to talk about WNL's BE's posthoc Power analysis. For a long time, WNL's Power is considered as statistical NONSENSE, see: HS's presentations such as this page 26, this page 12, and this page 22. Of course, I agree with our HS's opinion on this issue largely because of the strange method used by WNL for Power calculation, and the strange results as well. Actually, I have consulted a medical statistician who is professional at design and analysis of clinical trials. He told me he usually do power analysis to determine sample size before starting a trial, but seldom calc posthoc power, he also told me that even if the posthoc power is a little lower than designed or expected power, it is no necessary to do additional trial to increase the sample size and expect to increase the power. Consistent with HS’s and Potvin's opinions. Regardless of the necessity, let's first look inside into WNL to see how it calc power. From PNX WNL User Guide P341342 Power Power is the posthoc probability of detecting a difference greater than or equal to a specified percentage of the reference least squares mean. In general, Power = 1 – (probability of a type II error) In the bioequivalence calculations, the hypotheses being tested are: H_{0}: RefLSM = TestLSM For the notransform case, the power is the probability of rejecting H_{0} given: TestLSM – RefLSM  ≥ fractionToDetect × RefLSM For lntransform, and data already lntransformed, this changes to: TestLSM – RefLSM  ≥ – ln(1 – fractionToDetect), and similarly for log10transform and data already log10transformed. For the sake of illustration, assume fractionToDetect = 0.2, RefLSM>0, and no transform was done on the data. Also the maximum probability of rejecting H_{0} occurs in the case of equality, TestLSM – RefLSM = 0.2×RefLSM. So: Power = Pr (rejecting H_{0} at the alpha level given Difference = 0.2×RefLSM) Let: t_{1} = t_{(1α/2),df} – 0.2×RefLSM / DiffSE Then: Power = Pr(t_{stat} > t_{1} given Difference = 0.2×RefLSM) Note that t_{stat} is Tdistributed. Let p_{1} be the pvalue associated with t_{1} (the area in the tail), and let p_{2} be the pvalue associated with t_{2} (the area in the tail; note that p_{2} may be negligible). Then: Power = 1 – (p_{1} – p_{2}) From PNX WNL User Guide P341342 regretfully, WNL doesn't give us the formula when data is ln=transformed before BE analysis. Before we start to validate WNL's formula, let me show my first puzzle: As pointed by above WNL's user guide, it is obviouse that WNL is calculating the Power for TOST, but in WNL's ASCII output WNL gives us Power of ANOVA for Confidence Level 90.00 for ANOVA or for TOST? It is a question. Now, let me validate WNL's formula using Chow and Liu's dataset. For original data without transformation WNL's result is Power of ANOVA for Confidence Level 90.00 My formula and result is: t1 = t (1α/2),df – (1  0.8)×RefLSM / DiffSE Power Analysis For lntransformed data WNL's result is Power of ANOVA for Confidence Level 90.00 My formula and result is: t1 = t (1α/2),df + Ln(0.8) / DiffSE Power Analysis To ensure the sufficient precision of my calc, all the above calculations were done using MS Excel 2010 which is of much more higher precision than MS Excel 2000, 2003 and 2007 when Tinv() or Tdist() is used. For comparison: Software Tinv(0.1,22) It should be noted that the formula I used for Lntransformed data is of no reference, only obtained by my guess, but it works well specially to obtain WNL's results. I don't know whether it is correct or not and don't know WNL is correct or not. If we using PowerTOST as a gold standard for power calc of TOST, all WNL's results are and totally wrong. It seems that WNL's posthoc power calc doesn't directly use Observed Diff or Observed Ratio or intraCV, only indirectly uses DiffSE and Expected Diff or Ratio such as 0.8, 1,2, 0.8, 1.25. Dear all, True or False of WNL's method? Dear HS, have you reached Berlin and meet our D. Labes? I need your insight and comments on this issue. 
yicaoting Regular NanKing, China, 20111108 13:55 (edited by yicaoting on 20111108 14:11) @ yicaoting Posting: # 7637 Views: 5,876 

go on validating WNL's and my formula using unbalanced data Chow and Liu's famous data, delete Subject # 23 and 24's data both in period 1 and period 2. For original data WNL's result is My result is Power Analysis For lntransformed data WNL's result is Power of ANOVA for Confidence Level 90.00 My result is Power Analysis also same at level of 0.0001. Seems my formula works as WNL's inside hiden formula. The level of the precision is interestingly same to that of BE's 90% CI. go on validating WNL's and my formula using full data Original data, use 93% CI, 20% diff to detect All the results are satisfactory. Seems I have touched into WNL's black box. Edit: Merged two posts. [Helmut] 
Helmut Hero Vienna, Austria, 20111108 14:06 @ yicaoting Posting: # 7638 Views: 5,923 

Dear yicaoting, thanks for stepping into murky waters. » It seems that WNL's posthoc power calc doesn't directly use Observed Diff or Observed Ratio or intraCV, only indirectly uses DiffSE and Expected Diff or Ratio such as 0.8, 1,2, 0.8, 1.25. Dear all, True or False of WNL's method? Fascinating. I’m a little bit busy now, but will have a look as soon as possible. » Dear HS, have you reached Berlin and meet our D. Labes?
— Cheers, Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. ☼ Science Quotes 
d_labes Hero Berlin, Germany, 20111110 08:49 @ yicaoting Posting: # 7646 Views: 5,773 

Dear yicaoting, dear all, » From PNX WNL User Guide P341342Power » ... » In the bioequivalence calculations, the hypotheses being tested are: » » H_{0}: RefLSM = TestLSM » H_{1}: RefLSM ≠ TestLSM » ... This explains all! These hypotheses are for a superiority test or test for difference! Thus the first sentence should correctly read "In the bioequivalence calculations, the hypotheses not being tested ..." In BE testing we are interested in testing the reversed hypotheses, i.e. the Null is inequivalence, the Alternative is equivalence RefLSM = TestLSM (within some reasonable chosen margins). This explains why the power given in WLN is higher then calculated by PowerTOST. It's well known that the superiority test has higher power than the equivalence or noninferiority test. The formulas given by yicaoting point in the same direction (power for a test of difference), but complicated by the fact that not the usual noncentral tdistribution is used in the power calculation but an approximation via 'shifted' central t distribution. I can't reproduce the numbers because I don't have RefLSM and the DiffSE. Yicaoting, be so kind to post them here. To answer the question » Dear all, True or False of WNL's method? I would state: Approximate correct for the problem stated but answers a question no one has asked . — Regards, Detlew 
Helmut Hero Vienna, Austria, 20111110 13:39 @ d_labes Posting: # 7650 Views: 5,770 

Dear Detlew, was really nice meeting you yesterday! BTW, fog at both Berlin’s and Vienna’s airports; arrived at home at midnight… » I can't reproduce the numbers because I don't have RefLSM and the DiffSE. Yicaoting, be so kind to post them here. Results form PHX/WNL’s BE output (standard model; subject(sequence) random): Full dataset (balanced; n_{RT}=n_{TR}=12) Untransformed LSMRef 82.559375 SERef 4.34005723555842 LSMTest 80.271875 SETest 4.34005723555842 LSMDiff 2.28750000000002 SEDiff 3.73326038107666 ‘Power’ 0.9935408 ln transformed LSMRef 4.37972882180257 SERef 0.05625992661764 LSMTest 4.35107672956901 SETest 0.05625992661764 LSMDiff 0.028652092233559 SEDiff 0.055693090418743 (corrected; see here – was 0.02865… )‘Power’ 0.9839865 Reduced dataset (imbalanced; 23/24 excluded: n_{RT}=10, n_{TR}=12) Untransformed LSMRef 83.9116666666666 SERef 4.59672942436518 LSMTest 81.5147916666666 SETest 4.59672942436518 LSMDiff 2.39687499999999 SEDiff 3.81747472994938 ‘Power’ 0.99266427 ln transformed LSMRef 4.39812486326424 SERef 0.059448124239987 LSMTest 4.35107672956901 SETest 0.059448124239987 LSMDiff 0.032751848318541 SEDiff 0.057311390736542 ‘Power’ 0.97882724 » I would state: Approximate correct for the problem stated but answers a question no one has asked . Great. — Cheers, Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. ☼ Science Quotes 
d_labes Hero Berlin, Germany, 20111110 15:42 @ Helmut Posting: # 7652 Views: 5,821 

Dear Helmut, » was really nice meeting you yesterday! Meeting You, yes it was, really ! » BTW, fog at both Berlin’s and Vienna’s airports; arrived at home at midnight… For that I feel sorry for you. Eventually we should had smoked to a lesser degree? Regarding the WLN power I must correct myself to a certain degree. Chow, Shao and Wang^{1)} call a test with the hypotheses pair a test of equality to distinguish it from a superiority test with the hypothesis pair with delta the superiority margin. Ok, I think this is semantics. These authors give for the crossover design the following formula for the power of the 'equality test' (have translated it in R notation): power = 1  pt(tcrit,df,ncp=eps*sqrt(2*n)/sigma) where eps is the true difference for which the power shall calculated and tcrit is the quantile of the central tdistribution to the confidence level 1alpha/2 with df = n2 degrees of freedom. Setting eps=0.2*LSMref and sigma/sqrt(2*n)=SEdiff and let b=0.2*LSMref/SEdiff we get
With the approximation of the noncentral tdistri via 'shifted' central tdistri according to
we obtain yicaotings formulas! Lets calculate via noncentral tdistribution using your given data: Full dataset, untransformed b = 4.42291 Seems to agree within the approximation used in WNL. At least the right order of magnitude. Full dataset, logtransformed b = 30.57179 Seem not to function! No agreement to the WNL results. Due to insufficient degree of approximation for large noncentrality or due to false formula? b = ln(0.8)/SEdiff = 7.788037 (yicaotings formula) Seems also not to function! This is surprising! yicaoting: Which SEdiff did you use? Reduced dataset is the homework for all . Sorry for all that numbers with insufficient decimals. ^{1)} Chow, Shao and Wang "Sample size calculations in clinical research" Marcel Dekker, New York, NY 2003 — Regards, Detlew 
Helmut Hero Vienna, Austria, 20111110 16:00 @ d_labes Posting: # 7653 Views: 5,730 

Dear Detlew, » » BTW, fog at both Berlin’s and Vienna’s airports; arrived at home at midnight… » For that I feel sorry for you. Eventually we should had smoked to a lesser degree? Right. Maybe some cloud condensation nuclei leaked out… Vienna is smoker’s territory anyhow and the airport is just 1.8 km from river Danube. Was a starry night when I arrived home. » Chow, Shao and Wang… » […] I think this is semantics. Me too. » Reduced dataset is the homework for all . Yes, sir. ASAP. — Cheers, Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. ☼ Science Quotes 
d_labes Hero Berlin, Germany, 20111111 08:55 @ Helmut Posting: # 7656 Views: 5,923 

Dear Helmut, seems there is an error / copy and paste mistake in your SEdiff of the full set, logtransformed. My beasty number cruncher _{} (Proc MIXED) gives: Least Squares Means So repeating the power calculations for the Full set, logtransformed : b = ln(0.8)/SEdiff = 4.006887 (yicaotings formula) Seems reasonable in accordance to WNL I think. So mystery resolved. Thanx to yicaoting . — Regards, Detlew 
Helmut Hero Vienna, Austria, 20111111 13:39 @ d_labes Posting: # 7659 Views: 5,675 

Dear Detlew! » seems there is an error / copy and paste mistake in your SEdiff of the full set, logtransformed. Oops! Clever enough to CtrlC–CtrlV, too stupid to select the right column. » My beasty number cruncher _{} (Proc MIXED) gives: » Standard » Effect Formulation _Formulation Estimate Error DF t Value » Formulation Referenc Test 0.02865 0.05569 22 0.51 PHX/WNL: LSMDiff 0.028652092233559 SEDiff 0.055693090418743 THX! — Cheers, Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. ☼ Science Quotes 
yicaoting Regular NanKing, China, 20111112 06:05 @ d_labes Posting: # 7661 Views: 5,651 

Dear d_labes, » yicaoting: Which SEdiff did you use? My Diff and DiffSE are Diff(TR) = 0.0286520922335587 DiffSE = 0.055693090418743 the same to those of WNL. 