Ben ★ 20120302 15:12 (3737 d 03:02 ago) Posting: # 8206 Views: 8,804 

Dear all, I read a bit about the power calculation methods (exact, approximation) and are wondering about some details (I couldn't find the answer already in old posts). On slide 31 from Helmut's lecture it's written that exact methods rely on AS 243 (or AS 184). I thought the exact method stands for solving the integral defined by Owens Q function (see also this post); but AS 243 (and AS 184) are algorithms to compute cumulative probabilities of the noncentral tdistribution (aren't they?) and hence are only applicable in the case where the approximation via noncentral tdistribution is being used. So one cannot talk about using the exact method and AS 243 in one sentence. Or am I wrong here? Also, the nQuery v7 user manual (Appendix 75, page 153) says that an algorithm due to Owen is used in order to calculate the power, therefore on slide 33 I don't understand why the algorithm from nQuery is "AS 184" (well, if the algorithm from Owen is exactly AS 184, then it's ok...). Coming from another point of view it's getting clear why it cannot be the algorithm of Owen (like in the row above: Diletti et al (1991)): the sample sizes for example in case of CV=0.075 from Diletti et al (1991) and nQuery Advisor 7 do not match. So what about the user manual...? Another thing is: AS 184 is older than AS 243, but is it worse? For example if the true ratio=1, CV=0.075 and n=4 (see also this post) the exact method from PowerTOST gives a power of 0.7290143. nQuery 7 (see the table in the post just mentioned) gives 71.559% whereas FARTSSIE 1.6 (with AS 243) gives 66.674%. The result from nQuery is closer to the "exact" result, although it uses an older algorithm. Thank you in advance for all your thoughts on that. Ben PS Dear d_labes, while playing around with PowerTOST I saw the following happen:
> sampleN.TOST(theta0=1, CV=0.075, targetpower=0.6, method="noncentral", details=FALSE) There is this extra line (shown in red) stating the sample size and power (for all methods and details=FALSE). I guess this is just a bug... 
Helmut ★★★ Vienna, Austria, 20120302 17:21 (3737 d 00:53 ago) @ Ben Posting: # 8208 Views: 8,722 

Dear Ben! » On slide 31 from Helmut's lecture it's written that exact methods rely on AS 243 (or AS 184). I thought the exact method stands for solving the integral defined by Owens Q function (see also this post); but AS 243 (and AS 184) are algorithms to compute cumulative probabilities of the noncentral tdistribution (aren't they?) and hence are only applicable in the case where the approximation via noncentral tdistribution is being used. So one cannot talk about using the exact method and AS 243 in one sentence. Well, I could. I must confess that my wording might be confusing. I used ‘exact’ in a sense of numerically approximating Owen’s solution. » Or am I wrong here? Not at all. Though Owen gave the solution in his 1965 paper – the differences between the two definite integrals cannot be solved explicitly. Hence numeric methods have to be applied (see the nice numbers in logΓ section of the FORTRAN90source of AS 234). » Also, the nQuery v7 user manual (Appendix 75, page 153) says that an algorithm due to Owen is used in order to calculate the power, therefore on slide 33 I don't understand why the algorithm from nQuery is "AS 184" Blast! Where did I get this from? If I recall it right once there was a paper on Statsol’s website. Gone yet. I can also no confirm which version is implemented in FARTSSIE… No version is given in the VBAcode. My knowlege of FORTRAN is better than of VBA; maybe I hack myself through. There are some differences in the implementation, but I cannot say yet whether the VBAcode is more close to 184 or 234. Where did I get all these algonumbers from? Have to chimneysweep my future slides. » (well, if the algorithm from Owen is exactly AS 184, then it's ok...). Owen’s method is no algo. Unfortunately nothing is stated in the manual (neither in v7 nor in v5). » Coming from another point of view it's getting clear why it cannot be the algorithm of Owen (like in the row above: Diletti et al (1991)): the sample sizes for example in case of CV=0.075 from Diletti et al (1991) and nQuery Advisor 7 do not match. So what about the user manual...? Note that nQuery always give the sample size in integers per sequence. Therefore in a 2×2 crossover the output is 3 which gives a total of 6. Diletti et al. in their Table 1 give also 6, but in the heading the additional statement “Calculated odd sample sizes have been rounded up and are given in italics.” – which was the case for power 80%, PE 0.95, CV 7.5%. The respective rows in my table (slide 33) give the unrounded (odd) numbers, if applicable. » Another thing is: AS 184 is older than AS 243, but is it worse? Good question. Next question. » For example if the true ratio=1, CV=0.075 and n=4 (see also this post) the exact method from PowerTOST gives a power of 0.7290143. nQuery 7 (see the table in the post just mentioned) gives 71.559% whereas FARTSSIE 1.6 (with AS 243) gives 66.674%. The result from nQuery is closer to the "exact" result, although it uses an older algorithm. Interesting! But as I said above right now I cannot confirm which algo R. Lenth implemented into FARTSSIE. David’s statement ‘Dr. Russel Lenth generously provided the library subroutines to calculate noncentral distributions (NCt)’ is not telling. See also the end of this post. Maybe VBA runs into trouble at T/R=1; have to dig out Dieter’s paper (can’t promise – piles are high). — Diftor heh smusma 🖖 _{} Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes 
Ben ★ 20120302 18:14 (3737 d 00:00 ago) @ Helmut Posting: # 8209 Views: 7,470 

Dear Helmut, Thanks for the quick reply! » » On slide 31 from Helmut's lecture it's written that exact methods rely on AS 243 (or AS 184). I thought the exact method stands for solving the integral defined by Owens Q function (see also this post); but AS 243 (and AS 184) are algorithms to compute cumulative probabilities of the noncentral tdistribution (aren't they?) and hence are only applicable in the case where the approximation via noncentral tdistribution is being used. So one cannot talk about using the exact method and AS 243 in one sentence. » » Well, I could. I must confess that my wording might be confusing. I used ‘exact’ in a sense of numerically approximating Owen’s solution. Ok, got it. Thanks for clarifiying. » » Or am I wrong here? » » Not at all. Puh! Good » Owen’s method is no algo. Unfortunately nothing is stated in the manual (neither in v7 nor in v5). I don't have the paper from Owen, so I cannot tell. But thanks for letting me know; so the wording "an algorithm due to Owen" in the manual is not the best one... Yes, unfortunately it is not clear what nQuery does. » » Another thing is: AS 184 is older than AS 243, but is it worse? » » Good question. Next question. » Interesting! But as I said above right now I cannot confirm which algo R. Lenth implemented into FARTSSIE. David’s statement ‘Dr. Russel Lenth generously provided the library subroutines to calculate noncentral distributions (NCt)’ is not telling. See also the end of this post. Maybe VBA runs into trouble at T/R=1; have to dig out Dieter’s paper (can’t promise – piles are high). Ok, I got your point. As an alternative to FARTSSIE, isn't R using AS 243 for sure? At least this result (meaning 66.674%) is confirmed by using PowerTOST with method=noncentral. 
Helmut ★★★ Vienna, Austria, 20120302 19:38 (3736 d 22:36 ago) @ Ben Posting: # 8210 Views: 7,610 

Dear Ben! » » Owen’s method is no algo. […] » I don't have the paper from Owen, so I cannot tell. Expect some “Mehl”. » […] so the wording "an algorithm due to Owen" in the manual is not the best one... Yes, unfortunately it is not clear what nQuery does. Yep (twice). » Ok, I got your point. As an alternative to FARTSSIE, isn't R using AS 243 for sure? At least this result (meaning 66.674%) is confirmed by using PowerTOST with method=noncentral. Right. From R’s help(qt) :For the noncentral case of BTW, we have already shown that SAS and R get identical values based on the noncentral t – even for extreme values (#4297, part of monsterthread #4291). If you are a Caficionado, sourcecode here. @Anders: See above link at ‘Related Data and Programs’. Maybe useful in your powersims. Remember Detlew’s remark? — Diftor heh smusma 🖖 _{} Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes 
Ben ★ 20120303 13:35 (3736 d 04:39 ago) @ Helmut Posting: # 8214 Views: 7,434 

Dear Helmut, » The noncentral case is done by inversion. » BTW, we have already shown that SAS and R get identical values based on the noncentral t – even for extreme values (#4297, part of monsterthread #4291). Yes. The only thing I wanted to conclude here is that nQuery does not use AS 243 and I wanted so see whether nQuery uses AS 184 or not. For the latter one would need a C lib or something similar (yeh I like C ). Btw, I don't see the need anymore for testing AS 243 since it's used in R. Anyhow, in the post you just mentioned (actually #441) I saw you saying that "According to Dieter Hauschke Owen's exact method is implemented in nQuery Advisor." In fact in their book this is also stated. It seems all problems are solved (this is "confirmed" by the comparison in #4255.)... but let me ask two more things. First, there still are some discrepancies, e.g. in case of T/R=1, CV=7.5% (see e.g. #4266): for n=4 power.TOST "exact" gives 0.7290143 whereas nQuery gives 0.71559 (for n=6: 0.9869919 vs. 0.97943). I guess it's just not the same exact method that is implemented? (rounding?) Secondly, on your lecture slide 33 (see above) you wrote Diletti et al. (1991) uses method "noncentr. t" and algorithm "Owen's Q". Shouldn't it then be method "exact"? 
Helmut ★★★ Vienna, Austria, 20120303 15:34 (3736 d 02:40 ago) @ Ben Posting: # 8215 Views: 7,508 

Dear Ben! » The only thing I wanted to conclude here is that nQuery does not use AS 243 and I wanted so see whether nQuery uses AS 184 or not. Actually we simply don’t know. Couldn’t find a hint in the code… Are you good in reverseengineering? » For the latter one would need a C lib or something similar (yeh I like C ). Rather a decompiler. » […] but let me ask two more things. » First, there still are some discrepancies, e.g. in case of T/R=1, CV=7.5% (see e.g. #4266): for n=4 power.TOST "exact" gives 0.7290143 whereas nQuery gives 0.71559 (for n=6: 0.9869919 vs. 0.97943). I guess it's just not the same exact method that is implemented? (rounding?) No idea. » Secondly, on your lecture slide 33 (see above) you wrote Diletti et al. (1991) uses method "noncentr. t" and algorithm "Owen's Q". Shouldn't it then be method "exact"? Well, cough. This presentation is history. I’m not a fan of rewriting the past. In future presentations I will try to make it more clear…
— Diftor heh smusma 🖖 _{} Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes 
d_labes ★★★ Berlin, Germany, 20120313 12:01 (3726 d 06:13 ago) @ Ben Posting: # 8260 Views: 7,221 

Dear Ben, » First, there still are some discrepancies, e.g. in case of T/R=1, CV=7.5% (see e.g. #4266): for n=4 power.TOST "exact" gives 0.7290143 whereas nQuery gives 0.71559 (for n=6: 0.9869919 vs. 0.97943). I guess it's just not the same exact method that is implemented? (rounding?) These discrepancies may be due to not at least hidden bugs! They are too great in magnitude to be simply the result of rounding errors. I have implemented the evaluation of the distinct integral the Qfunction is based on for simplicity and in analogy to the function power.equivalence.md() of package MBESS using the Rfunction integrate() from package stats . This is a little bit "black box programming" because I have no precise idea what integrate() really does and if there are circumstances where it can fail like all numeric integration routines.To check my implementation in package PowerTOST I have coded the algorithm given by Owen itself (repeated integration by parts) in a function OwensQOwen() .Using the calls that results if one tries to compute the power for T/R=1, CV=7.5% and n=4 the following results are obtained: df=2 Boing! Totally consistence of the values from both implementations . (Red is rounding to the results from the call of power.TOST() with full internal precision of tvalue, delta1, delta2 and R).The same coincidence is observed with T/R=1, CV=7.5% and n=6 with power=0.9869919 via OwensQOwen() .The deduction in which software there is eventually a bug or a flawed implementation is left to you . BTW: OwensQOwen() code is available upon request.— Regards, Detlew 
Ben ★ 20120313 21:52 (3725 d 20:22 ago) @ d_labes Posting: # 8269 Views: 7,145 

Dear all, thank you very much for the great discussion and the valuable comments. I guess, to some extent, it will remain unclear what is really going on... Best regards, Ben 
ElMaestro ★★★ Denmark, 20120304 21:56 (3734 d 20:18 ago) @ Helmut Posting: # 8217 Views: 7,396 

Hi Ben and HS, » Interesting! But as I said above right now I cannot confirm which algo R. Lenth implemented into FARTSSIE. David’s statement ‘Dr. Russel Lenth generously provided the library subroutines to calculate noncentral distributions (NCt)’ is not telling. From a quick look at the code snippets, This code (in particular the function called 'tnc') corresponds to the 'NCt' function of Fartssie's module 1. I could not find an implementation of AS 184 for comparison (anyone has a link? I googled a little but did not do much else to find it). I would say that to my untrained eye it looks like Fartssie emulates some bits of the AS 243 routines. — Pass or fail! ElMaestro 
Helmut ★★★ Vienna, Austria, 20120305 01:57 (3734 d 16:17 ago) @ ElMaestro Posting: # 8218 Views: 7,426 

Dear ElMeastro! » I could not find an implementation of AS 184 for comparison (anyone has a link? I googled a little but did not do much else to find it). If you have 20 bucks to spare: http://www.jstor.org/pss/2348011 » I would say that to my untrained eye it looks like Fartssie emulates some bits of the AS 243 routines. Yep. — Diftor heh smusma 🖖 _{} Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes 
d_labes ★★★ Berlin, Germany, 20120305 15:28 (3734 d 02:47 ago) @ ElMaestro Posting: # 8220 Views: 7,413 

My very dear Pirat, » I could not find an implementation of AS 184 for comparison (anyone has a link? I googled a little but did not do much else to find it). Found http://lib.stat.cmu.edu/apstat/184 also via > "AS 184 noncentral tdistribution statlib" > 4th entry . Seems in FORTRAN. But the title is somewhat deviating from noncentral tdistribution . — Regards, Detlew 
d_labes ★★★ Berlin, Germany, 20120305 15:17 (3734 d 02:57 ago) @ Ben Posting: # 8219 Views: 7,305 

Dear Ben, » There is this extra line (shown in red) stating the sample size and power (for all methods and details=FALSE). I guess this is just a bug... Yep. Thanks to call my attention to this. Fixed. — Regards, Detlew 