Bioequivalence and Bioavailability Forum

Ben
★

2012-03-02 16:12
(4805 d 22:59 ago)

Posting: # 8206
Views: 10,472

Exact and approximate method [Power / Sample Size]

Dear all,

I read a bit about the power calculation methods (exact, approximation) and are wondering about some details (I couldn't find the answer already in old posts).
On slide 31 from Helmut's lecture it's written that exact methods rely on AS 243 (or AS 184). I thought the exact method stands for solving the integral defined by Owens Q function (see also this post); but AS 243 (and AS 184) are algorithms to compute cumulative probabilities of the noncentral t-distribution (aren't they?) and hence are only applicable in the case where the approximation via noncentral t-distribution is being used. So one cannot talk about using the exact method and AS 243 in one sentence. Or am I wrong here?

Also, the nQuery v7 user manual (Appendix 7-5, page 153) says that an algorithm due to Owen is used in order to calculate the power, therefore on slide 33 I don't understand why the algorithm from nQuery is "AS 184" (well, if the algorithm from Owen is exactly AS 184, then it's ok...). Coming from another point of view it's getting clear why it cannot be the algorithm of Owen (like in the row above: Diletti et al (1991)): the sample sizes for example in case of CV=0.075 from Diletti et al (1991) and nQuery Advisor 7 do not match. So what about the user manual...?

Another thing is: AS 184 is older than AS 243, but is it worse? For example if the true ratio=1, CV=0.075 and n=4 (see also this post) the exact method from PowerTOST gives a power of 0.7290143. nQuery 7 (see the table in the post just mentioned) gives 71.559% whereas FARTSSIE 1.6 (with AS 243) gives 66.674%. The result from nQuery is closer to the "exact" result, although it uses an older algorithm.

Thank you in advance for all your thoughts on that.
-Ben

PS
Dear d_labes, while playing around with PowerTOST I saw the following happen:


> sampleN.TOST(theta0=1, CV=0.075, targetpower=0.6, method="noncentral", details=FALSE)



+++++++++++ Equivalence test - TOST +++++++++++

            Sample size estimation

-----------------------------------------------

Study design:  2x2 crossover 

log-transformed data (multiplicative model)



alpha = 0.05, target power = 0.6

BE margins        = 0.8 ... 1.25 

Null (true) ratio = 1,  CV = 0.075

4   0.666742 



Sample size (total)

 n     power

4   0.666742 



Approximate power calculation with

non-central t-distribution.

There is this extra line (shown in red) stating the sample size and power (for all methods and details=FALSE). I guess this is just a bug...

Helmut
★★★

Vienna, Austria,
2012-03-02 18:21
(4805 d 20:51 ago)

@ Ben
Posting: # 8208
Views: 10,236

Sorry for the confusion caused

Post reply

Dear Ben!

❝ On slide 31 from Helmut's lecture it's written that exact methods rely on AS 243 (or AS 184). I thought the exact method stands for solving the integral defined by Owens Q function (see also this post); but AS 243 (and AS 184) are algorithms to compute cumulative probabilities of the noncentral t-distribution (aren't they?) and hence are only applicable in the case where the approximation via noncentral t-distribution is being used. So one cannot talk about using the exact method and AS 243 in one sentence.

Well, I could. :-D

I must confess that my wording might be confusing. I used ‘exact’ in a sense of numerically approximating Owen’s solution.

❝ Or am I wrong here?

Not at all. Though Owen gave the solution in his 1965 paper – the differences between the two definite integrals cannot be solved explicitly. Hence numeric methods have to be applied (see the nice numbers in log-Γ section of the FORTRAN90-source of AS 234).

❝ Also, the nQuery v7 user manual (Appendix 7-5, page 153) says that an algorithm due to Owen is used in order to calculate the power, therefore on slide 33 I don't understand why the algorithm from nQuery is "AS 184"

Blast! Where did I get this from? If I recall it right once there was a paper on Statsol’s website. Gone yet. :-(

I can also no confirm which version is implemented in FARTSSIE… No version is given in the VBA-code. My knowlege of FORTRAN is better than of VBA; maybe I hack myself through. There are some differences in the implementation, but I cannot say yet whether the VBA-code is more close to 184 or 234. Where did I get all these algo-numbers from? Have to chimney-sweep my future slides.

❝ (well, if the algorithm from Owen is exactly AS 184, then it's ok...).

Owen’s method is no algo. Unfortunately nothing is stated in the manual (neither in v7 nor in v5).

❝ Coming from another point of view it's getting clear why it cannot be the algorithm of Owen (like in the row above: Diletti et al (1991)): the sample sizes for example in case of CV=0.075 from Diletti et al (1991) and nQuery Advisor 7 do not match. So what about the user manual...?

Note that nQuery always give the sample size in integers per sequence. Therefore in a 2×2 cross-over the output is 3 which gives a total of 6. Diletti et al. in their Table 1 give also 6, but in the heading the additional statement “Calculated odd sample sizes have been rounded up and are given in italics.” – which was the case for power 80%, PE 0.95, CV 7.5%. The respective rows in my table (slide 33) give the unrounded (odd) numbers, if applicable.

❝ Another thing is: AS 184 is older than AS 243, but is it worse?

Good question. Next question.

❝ For example if the true ratio=1, CV=0.075 and n=4 (see also this post) the exact method from PowerTOST gives a power of 0.7290143. nQuery 7 (see the table in the post just mentioned) gives 71.559% whereas FARTSSIE 1.6 (with AS 243) gives 66.674%. The result from nQuery is closer to the "exact" result, although it uses an older algorithm.

Interesting! But as I said above right now I cannot confirm which algo R. Lenth implemented into FARTSSIE. David’s statement ‘Dr. Russel Lenth generously provided the library subroutines to calculate non-central distributions (NCt)’ is not telling. See also the end of this post. Maybe VBA runs into trouble at T/R=1; have to dig out Dieter’s paper (can’t promise – piles are high).

—
Dif-tor heh smusma 🖖🏼 Довге життя Україна!
Helmut Schütz

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

Ben
★

2012-03-02 19:14
(4805 d 19:58 ago)

@ Helmut
Posting: # 8209
Views: 8,866

Thanks for clarifying

Post reply

Dear Helmut,

Thanks for the quick reply!

❝ ❝ On slide 31 from Helmut's lecture it's written that exact methods rely on AS 243 (or AS 184). I thought the exact method stands for solving the integral defined by Owens Q function (see also this post); but AS 243 (and AS 184) are algorithms to compute cumulative probabilities of the noncentral t-distribution (aren't they?) and hence are only applicable in the case where the approximation via noncentral t-distribution is being used. So one cannot talk about using the exact method and AS 243 in one sentence.

❝

❝ Well, I could. :-D I must confess that my wording might be confusing. I used ‘exact’ in a sense of numerically approximating Owen’s solution.

Ok, got it. Thanks for clarifiying.

❝ ❝ Or am I wrong here?

❝

❝ Not at all.

Puh! Good

❝ Owen’s method is no algo. Unfortunately nothing is stated in the manual (neither in v7 nor in v5).

I don't have the paper from Owen, so I cannot tell. But thanks for letting me know; so the wording "an algorithm due to Owen" in the manual is not the best one... Yes, unfortunately it is not clear what nQuery does.

❝ ❝ Another thing is: AS 184 is older than AS 243, but is it worse?

❝

❝ Good question. Next question.

❝ Interesting! But as I said above right now I cannot confirm which algo R. Lenth implemented into FARTSSIE. David’s statement ‘Dr. Russel Lenth generously provided the library subroutines to calculate non-central distributions (NCt)’ is not telling. See also the end of this post. Maybe VBA runs into trouble at T/R=1; have to dig out Dieter’s paper (can’t promise – piles are high).

Ok, I got your point. As an alternative to FARTSSIE, isn't R using AS 243 for sure? At least this result (meaning 66.674%) is confirmed by using PowerTOST with method=noncentral.

Helmut
★★★

Vienna, Austria,
2012-03-02 20:38
(4805 d 18:34 ago)

@ Ben
Posting: # 8210
Views: 9,028

Owen’s Q (Anders, are you there?)

Post reply

Dear Ben!

❝ ❝ Owen’s method is no algo. […]

❝ I don't have the paper from Owen, so I cannot tell.

Expect some “Mehl”. ;-)

❝ […] so the wording "an algorithm due to Owen" in the manual is not the best one... Yes, unfortunately it is not clear what nQuery does.

Yep (twice).

❝ Ok, I got your point. As an alternative to FARTSSIE, isn't R using AS 243 for sure? At least this result (meaning 66.674%) is confirmed by using PowerTOST with method=noncentral.

Right. From R’s help(qt):

For the non-central case of pt based on a C translation of
Lenth, R. V. (1989). Algorithm AS 243 — Cumulative distribution function of the non-central t distribution, Applied Statistics 38, 185–189.
This computes the lower tail only, so the upper tail suffers from cancellation and a warning will be given when this is likely to be significant.
[…]
The non-central case is done by inversion.

BTW, we have already shown that SAS and R get identical values based on the non-central t – even for extreme values (#4297, part of monster-thread #4291).

If you are a C-aficionado, source-code here.

@Anders: See above link at ‘Related Data and Programs’. Maybe useful in your power-sims. Remember Detlew’s remark?

—
Dif-tor heh smusma 🖖🏼 Довге життя Україна!
Helmut Schütz

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

Ben
★

2012-03-03 14:35
(4805 d 00:36 ago)

@ Helmut
Posting: # 8214
Views: 8,851

Owen’s Q

Post reply

Dear Helmut,

❝ The non-central case is done by inversion.

❝ BTW, we have already shown that SAS and R get identical values based on the non-central t – even for extreme values (#4297, part of monster-thread #4291).

Yes. The only thing I wanted to conclude here is that nQuery does not use AS 243 and I wanted so see whether nQuery uses AS 184 or not. For the latter one would need a C lib or something similar (yeh I like C :-)

). Btw, I don't see the need anymore for testing AS 243 since it's used in R.

Anyhow, in the post you just mentioned (actually #441) I saw you saying that "According to Dieter Hauschke Owen's exact method is implemented in nQuery Advisor." In fact in their book this is also stated. It seems all problems are solved (this is "confirmed" by the comparison in #4255.)... but let me ask two more things.
First, there still are some discrepancies, e.g. in case of T/R=1, CV=7.5% (see e.g. #4266): for n=4 power.TOST "exact" gives 0.7290143 whereas nQuery gives 0.71559 (for n=6: 0.9869919 vs. 0.97943). I guess it's just not the same exact method that is implemented? (rounding?)
Secondly, on your lecture slide 33 (see above) you wrote Diletti et al. (1991) uses method "noncentr. t" and algorithm "Owen's Q". Shouldn't it then be method "exact"?

Helmut
★★★

Vienna, Austria,
2012-03-03 16:34
(4804 d 22:38 ago)

@ Ben
Posting: # 8215
Views: 8,895

Exact or not, that‘s the question

Post reply

Dear Ben!

❝ The only thing I wanted to conclude here is that nQuery does not use AS 243 and I wanted so see whether nQuery uses AS 184 or not.

Actually we simply don’t know. Couldn’t find a hint in the code… Are you good in reverse-engineering?

❝ For the latter one would need a C lib or something similar (yeh I like C :-) ).

Rather a decompiler. ;-)

❝ […] but let me ask two more things.

❝ First, there still are some discrepancies, e.g. in case of T/R=1, CV=7.5% (see e.g. #4266): for n=4 power.TOST "exact" gives 0.7290143 whereas nQuery gives 0.71559 (for n=6: 0.9869919 vs. 0.97943). I guess it's just not the same exact method that is implemented? (rounding?)

No idea.

❝ Secondly, on your lecture slide 33 (see above) you wrote Diletti et al. (1991) uses method "noncentr. t" and algorithm "Owen's Q". Shouldn't it then be method "exact"?

Well, cough. This presentation is history. I’m not a fan of rewriting the past.
In future presentations I will try to make it more clear…

Strictly speaking the ‘exact’ method based on Owen’s Q is also numeric approximation of the difference of two integrals of the bivariate noncentral t (which are not solvable in closed form).
Methods based on the noncentral t (approximating the above).
Methods based on the shifted t (approximating the above).

It’s always difficult what we define (or accept) as ‘exact’. F.i. in PK we can only calculate t_max for a one-compartment open model, extravascular administration (solve the root of the first derivate of the function). For more than one compartments the first derivate is not accessible in closed form. Nevertheless we can obtain the maximum (and its time) at any wanted numeric precision by an iterative procedure – even by means of a pocket calculator. Exact or not?

—
Dif-tor heh smusma 🖖🏼 Довге життя Україна!
Helmut Schütz

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

d_labes
★★★

Berlin, Germany,
2012-03-13 13:01
(4795 d 02:11 ago)

@ Ben
Posting: # 8260
Views: 8,646

Owen’s Q implementation issues

Post reply

Dear Ben,

These discrepancies may be due to not at least hidden bugs! They are too great in magnitude to be simply the result of rounding errors.

I have implemented the evaluation of the distinct integral the Q-function is based on for simplicity and in analogy to the function power.equivalence.md() of package MBESS using the R-function integrate() from package stats. This is a little bit "black box programming" because I have no precise idea what integrate() really does and if there are circumstances where it can fail like all numeric integration routines.

To check my implementation in package PowerTOST I have coded the algorithm given by Owen itself (repeated integration by parts) in a function OwensQOwen().

Using the calls that results if one tries to compute the power for T/R=1, CV=7.5% and n=4 the following results are obtained:

df=2

tvalue= 2.919986

delta1= 4.213542

delta2= -4.213542

R= 2.040712

p1 <- OwensQ(df, +tvalue, delta1, 0, R) # PowerTOST implementation

p2 <- OwensQ(df, -tvalue, delta2, 0, R)

power <- p2 - p1

-- gives

p1= 0.07316842

p2= 0.8021825

power = 0.7290141



p1 <- OwensQOwen(df, +tvalue, delta1, 0, R)  #Owen's integration by parts

p2 <- OwensQOwen(df, -tvalue, delta2, 0, R)

power <- p2 - p1

-- gives

p1= 0.07316842

p2= 0.8021825

power = 0.7290141

Boing! Totally consistence of the values from both implementations :cool:

.
(Red is rounding to the results from the call of power.TOST() with full internal precision of tvalue, delta1, delta2 and R).

The same coincidence is observed with T/R=1, CV=7.5% and n=6 with power=0.9869919 via OwensQOwen().

The deduction in which software there is eventually a bug or a flawed implementation is left to you :-D

.

BTW: OwensQOwen() code is available upon request.

—
Regards,

Detlew

Ben ★ 2012-03-13 22:52 (4794 d 16:19 ago) @ d_labes Posting: # 8269 Views: 8,557	Owen’s Q implementation issues Post reply
	Dear all, thank you very much for the great discussion and the valuable comments. I guess, to some extent, it will remain unclear what is really going on... Best regards, Ben

ElMaestro
★★★

Denmark,
2012-03-04 22:56
(4803 d 16:15 ago)

@ Helmut
Posting: # 8217
Views: 8,806

Sorry for the confusion caused

Post reply

Hi Ben and HS,

From a quick look at the code snippets, This code (in particular the function called 'tnc') corresponds to the 'NCt' function of Fartssie's module 1.
I could not find an implementation of AS 184 for comparison (anyone has a link? I googled a little but did not do much else to find it). I would say that to my untrained eye it looks like Fartssie emulates some bits of the AS 243 routines.

—
Pass or fail!
ElMaestro

Helmut
★★★

Vienna, Austria,
2012-03-05 02:57
(4803 d 12:15 ago)

@ ElMaestro
Posting: # 8218
Views: 8,852

Sorry for the confusion caused

Post reply

Dear ElMeastro!

❝ I could not find an implementation of AS 184 for comparison (anyone has a link? I googled a little but did not do much else to find it).

If you have 20 bucks to spare: http://www.jstor.org/pss/2348011

❝ I would say that to my untrained eye it looks like Fartssie emulates some bits of the AS 243 routines.

Yep.

—
Dif-tor heh smusma 🖖🏼 Довге життя Україна!
Helmut Schütz

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

d_labes ★★★ Berlin, Germany, 2012-03-05 16:28 (4802 d 22:44 ago) @ ElMaestro Posting: # 8220 Views: 8,811	AS 184 without wasting bucks Post reply
	My very dear Pirat, ❝ I could not find an implementation of AS 184 for comparison (anyone has a link? I googled a little but did not do much else to find it). Found http://lib.stat.cmu.edu/apstat/184 also via -> "AS 184 non-central t-distribution statlib" -> 4th entry . Seems in FORTRAN. But the title is somewhat deviating from non-central t-distribution . — Regards, Detlew

d_labes ★★★ Berlin, Germany, 2012-03-05 16:17 (4802 d 22:54 ago) @ Ben Posting: # 8219 Views: 8,741	PowerTOST_0.9-4 on CRAN now Post reply
	Dear Ben, ❝ There is this extra line (shown in red) stating the sample size and power (for all methods and details=FALSE). I guess this is just a bug... Yep. Thanks to call my attention to this. Fixed. — Regards, Detlew

Exact and approximate method [Power / Sample Size]

Sorry for the confusion caused

Thanks for clarifying

Owen’s Q (Anders, are you there?)

Owen’s Q

Exact or not, that‘s the question

Owen’s Q implementation issues

Owen’s Q implementation issues

Sorry for the confusion caused

Sorry for the confusion caused

AS 184 without wasting bucks

PowerTOST_0.9-4 on CRAN now