Bioequivalence and Bioavailability Forum • PowerTOST & beyond

Helmut
★★★

Vienna, Austria,
2010-06-11 20:08
(5434 d 21:37 ago)

Posting: # 5505
Views: 13,513

PowerTOST & beyond [🇷 for BE/BA]

Post reply

Dear all & especially D. Labes!
I played around with package PowerTOST and found a counterintuitive result. :confused:

I tried to get an 'optimal' total sample size (pilot + main study).
Assumptions: CV 25%-35%, T/R 0.95, 80% power -->

require(PowerTOST)

expsampleN.TOST(alpha = 0.05,

  targetpower = 0.80,

  theta1 = 0.80,

  theta2 = 1.25,

  diff = 0.95,

  CV = 0.25,     # variable: 0.25-0.35

  dfCV = 12-2,   # variable: 12-24

  alpha2 = 0.05,

  design = "2x2")

I varied the sample size of the pilot study in the range 12-24 and calculated the size of the main study. I got:


        CV = 25%    CV = 30%    CV = 35%

pilot  main total  main total  main total

 12     34   46     48   60     64   76

 14     34   48     46   60     62   76

 16     32   48     46   62     60   76

 18     32   50     44   62     58   76

 20     32   52     44   64     58   78

 22     32   54     44   66     58   80

 24     30   54     42   66     56   80

This puzzles me in two respects. Though the size of the main study decreases, if the size of the pilot increases (estimated CV more reliable), the estimated total size also increases. Fixed sample size for CV=25%-35% are 28/40/52. Another point is the difference between small and large pilots dependent on the CV. In my example for CV=25% the ratio of the total sample size (pilot 24/12) is 1.17, for CV=30% 1.10, and for CV=35% 1.05. From these results one could suspect that for higher CVs, the size of the pilot study becomes more and more irrelevant?!

Sancta Juliem, adsta!

Now I did it the 'old fashioned way' aka based on the _X²-distribution (Julious, Chow/Liu, Patterson/Jones, Gould), alpha 0.25 and got:

        CV = 25%    CV = 30%    CV = 35%

pilot  main total  main total  main total

 12     42   54     58   70     76   88

 14     40   54     56   70     72   86

 16     38   54     54   70     70   86

 18     38   56     52   70     70   86

 20     36   56     52   72     68   88

 22     36   58     50   72     66   88

 24     36   60     50   74     66   90

Higher numbers, but a similar pattern. The ratio of the total sample size (pilot 24/12) for CV=25 is 1.11, for CV=30% 1.06, and for CV=35% 1.02.
Shall I abandon my pet hypothesis and suggest "the smaller, the better" in the future?

—
Dif-tor heh smusma 🖖🏼 Довге життя Україна!
Helmut Schütz

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

d_labes
★★★

Berlin, Germany,
2010-06-15 17:25
(5431 d 00:20 ago)

(edited on 2010-06-16 08:28)
@ Helmut
Posting: # 5516
Views: 12,030

Small is beautiful

Post reply

Dear Helmut,

thanks for playing with that tool. If the man plays he is well :-D

.

Very interesting observation!

Let me expand your table:


         CV = 20%   CV = 25%    CV = 30%    CV = 35%

 pilot  main total main total  main total  main total

   6     32   38     50   56     68   74     96  102

   8     26   34     40   48     56   64     78   86

  10     24   34     36   46     52   62     72   82

  12     24   36     34   46     48   60     64   76

  14     22   36     34   48     46   60     62   76

  16     22   38     32   48     46   62     60   76

  18     22   40     32   50     44   62     58   76

  20     22   42     32   52     44   64     58   78

  22     22   44     32   54     44   66     58   80

  24     20   44     30   54     42   66     56   80

  ...    ...         ...         ...         ...

  inf    20  inf     28  inf     40  inf     52  inf

Is this what you expected to see?

Of course this tells us a well known story: Pilots with smaller than 12 subjects are not very useful.
But more then 24 subjects are also not recommendable with respect to the total sample size.
But this is only true if you take the uncertainty of the CV into account. Else you may end in an underpowered study.

For your second concern

❝ for higher CVs, the size of the pilot study becomes more and more irrelevant?!

I do not get exactly your point. "Small" and "large" pilot study is not so well defined here I think.

❝ Shall I abandon my pet hypothesis and suggest "the smaller, the better" in the future?

I think there is no sound reason to do so. Also there is some rumor out there: "Small is beautiful" :cool:

.

BTW: Be so kind and enlighten a non-latin educated person about Sancta.

—
Regards,

Detlew

Helmut
★★★

Vienna, Austria,
2010-06-15 18:54
(5430 d 22:51 ago)

@ d_labes
Posting: # 5524
Views: 12,003

Small is beautiful

Post reply

Dear D. Labes!

❝ Let me expand your table: ...

❝ Is this what you expected to see?

Not really. Of course the estimate approaches asymptotically the fixed value, but I expected some kind of award for performing a larger pilot study. A larger pilot gives me a 'better' estimate and therefore the size of the main study will be smaller. But if I add the sample sizes of pilot and main studies, I'm disapointed. See the end of my post.

❝ Of course this tells us a well known study: Pilots with smaller than 12 subjects are not very useful.

OK, right - common sense, supported by PowerTOST. It's interesting that there is a minimum total sample size - very useful!

❝ For your second concern

❝ ❝ for higher CVs, the size of the pilot study becomes more and more irrelevant?!

❝ I do not get exactly your point. "Small" and "large" pilot study is not so well defined here I think.

Let's look at the 30% CV example. If the pilot study had 12 subjects I would plan the main study for 48. In a 24 pilot I get a better estimate and plan the main in only 42. But I'm punished, because the total sample size (pilot+main) will be 66 instead of 60. This speaks against my pet hypothesis. I learned from your table that there seems to be an optimal pilot sample size (if the total size is concerned), namely for CV 20% 8-10, CV 25% 10-12, CV 30% 12-14, and CV 35% 14-16...

❝ BTW: Be so kind and enlighten a non-latin educated person about Sancta.

Invocationing Guru Stephen in English: Saint Juliuos, stay by me!

—
Dif-tor heh smusma 🖖🏼 Довге життя Україна!
Helmut Schütz

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

Alice
☆

2010-07-14 16:32
(5402 d 01:13 ago)

@ Helmut
Posting: # 5625
Views: 11,745

Small is beautiful

Post reply

Dear all.

For begin my post, I'm sorry if my english is not really good and if my question is not in the good thread.

I'm a student in biostatistics and I'll finish my studies in september. I'm doing my professional training in pharmaceutical company and I'm writtening my report. I've got some difficulties to understand tost power. I've looked at D.Labes R packages, but I don't understand how this function run and so I can't writte it in my report.

Maybe have you some reading for help me?

I've read lots of thread in BEBAC and I would to thanks all for your help and knowledge.

Helmut
★★★

Vienna, Austria,
2010-07-14 18:23
(5401 d 23:22 ago)

@ Alice
Posting: # 5628
Views: 11,945

Small is beautiful

Post reply

Dear Alice!

❝ […] I'm sorry if my english is not really good

No problem - we have just a few native speakers of English here.

❝ I've got some difficulties to understand tost power.

Power=1-beta, where in the framework of BE beta is the producer's risk to fail in demonstrating bioequivalence with a true bioequivalent formulation. Most companies try to plan for a sample size of ~90% (optimistic case: all assumptions on the CV, T/R-ratio hold, no drop-outs) in order to get ~80% if assumptions are violated (higher CV, T/R deviating more from unity, drop-outs).
You cannot calculate sample size directly, but only power based on fixed values: CV, T/R-ratio, alpha (generally 0.05), acceptance range (generally 0.80-1.25), sample size. For any combination of these values you get a power value. Now you increase the sample size until the calculated power is > the target power.
Example: alpha 0.05, beta 20% (target power: 1-beta=80%), T/R 0.95, CV_intra 20%. You start the iterative search with a sample size of 16 subjects and obtain:

  n    power

 16   73.54%

 17   76.51%

 18   79.12%

 19   81.43%

 20   83.47%

With 19 subjects you already exceed the target power of 80%. In a TR/RT 2×2×2 cross-over you will start with a balanced design (equal number of subjects in each sequence) - therefore you round up to the next even number 20 [N = n_TR (10) + n_RT (10); power 83.47%].

❝ Maybe have you some reading for help me?

Maybe you find one of my presentations useful. References are given at the end.

❝ I've looked at D.Labes R packages, but I don't understand how this function run and so I can't writte it in my report.

Have you tried help(PowerTOST) after loading the package?
The example above would be coded by means of sampleN.TOST()

sampleN.TOST(alpha = 0.05, targetpower = 0.8, logscale = TRUE,

theta1 = 0.8, theta2 = 1.25, diff = 0.95, CV = 0.2, design = "2x2",

exact = TRUE, print = TRUE, details = TRUE)

resulting in

+++++++++ Equivalence test - TOST +++++++++

          Sample size estimation

-------------------------------------------

Study design:  2x2 crossover 

Design characteristics:

df = n-2, design const. = 2, step = 2



log-transformed data (multiplicative model)



alpha = 0.05, target power = 0.8

BE margins        = 0.8 ... 1.25 

Null (true) ratio = 0.95,  CV = 0.2



Sample size search

 n     power

16   0.735413 

18   0.791240 

20   0.834680 



Exact power calculation with

Owen's Q functions.

Package sampleN.TOST() gives samples for balanced designs only (therefore no values for 17 and 19).

[image]

At CV 20% we plan the study with 20 subjects (red diamonds). Power (green lines) is 83.47%. If CV increases, we still can go with 20 subjects (although power decreases), until we reach CV 20.98%. Power would be 79.99% and we have to increase the sample size in order to stay >80%.
Another interesting point: The minimum sample size in most regulations is 12. This translates to a CV of 15.63% (power 80.02%). If we keep the sample size at 12 and the CV is even lower, it becomes more and more likely that we get a significant treatment effect (confidence interval does not include 100%). Might be problematic in Denmark. For formulations with very low variability (yes, I've seen a CV of 6%), 4 subjects would be enough (power 80.52%). If we run the study in 12, power will be 99.99993%. :cool:

According to ICH-E9 you should perform a sensitivity analysis in study planning. In that case power.TOST() helps. Let's assume that you planned the study with 20 subjects and want to know the power if T/R is 0.90 instead of 0.95...

power.TOST(alpha = 0.05, logscale = TRUE,

theta1 = 0.8, theta2 = 1.25, diff = 0.90,

CV = 0.2, n = 20, design = "2x2", exact = TRUE)

We get
[1] 0.5649986

Oh, that's bad.
Let's keep the T/R-ratio at 0.95 and increase the CV to 0.25 instead - we get
[1] 0.6430574

Generally power functions are quite flat on the top (~±5% from 100%, example plot), but drop off quite fast if we move away from 100%. The impact of CV is not so important. Drop outs have the least impact (we have seen already above that with 16 subjects power will still be 73.54%).

—
Dif-tor heh smusma 🖖🏼 Довге життя Україна!
Helmut Schütz

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

Alice
☆

2010-07-15 23:35
(5400 d 18:10 ago)

@ Helmut
Posting: # 5640
Views: 11,755

Small is beautiful

Post reply

❝ No problem - we have just a few native speakers of English here.

I suppose I'm understable, but I prefer to notice my english level :-)

. Thanks a lot for your comprehension and for taking your time.

Your example is clear and helps me to understand better basics about TOST power.

❝ Maybe you find one of my presentations useful. References are given at the end.

Thanks too for this link. I was already read your presentation and that helps me a lot. But I've got one other question about power and D.Labes function.
You had writte T/R = 0.95. With BE margins, we can have 0.80<T/R<1.25. So we need to test all T/R ratio with the function? Or maybe I had'nt understand something?

❝ Exact power calculation with

❝ Owen's Q functions.

I had understand how use D.Labes function (and thanks to you for your explanation! :-)

). Now, I hope I'll find some references for understand how works Owen's Q function (generally, it's not easy for a student to find an access to these references :-D

).

Thanks again for your post!!

Helmut
★★★

Vienna, Austria,
2010-07-16 02:14
(5400 d 15:31 ago)

@ Alice
Posting: # 5642
Views: 11,741

Small is beautiful

Post reply

Dear Alice!

❝ You had writte T/R = 0.95. With BE margins, we can have 0.80<T/R<1.25. So we need to test all T/R ratio with the function? Or maybe I had'nt understand something?

Well that's the expected mean deviation of test from reference of -5% (we are taking about average bioequivalence here). Most people use this value when they have no other information (like point estimates from previous studies, etc.) The new European Guideline on Bioequivalence states that formulations must not differ by more than 5% in their actual (=measured) contents. Power functions are slightly asymetrical in linear scale. At any given sample size and CV, power for T/R 0.95 is lower than for T/R 1.05 - but equal to T/R 0.95^-1.
For our yesterday's example (20 subjects) we get a power of 83.47% at T/R of 0.95 (and 0.95^-1=1.05263…), but 84.32% at 1.05.
When people sloppily talk about ±5% deviation of test from reference, it's therefore common practice to use 0.95 (not 1.05!) in order to get a conservative estimate of the sample size.

You can try the function with any value you want - but you will get
Err: ratio not between margins!
if ≤0.80 or ≥1.25. Play around with the power-function as well. What power do you expect close to the acceptance margins?

❝ Now, I hope I'll find some references for understand how works Owen's Q function (generally, it's not easy for a student to find an access to these references :-D ).

At least Biometrika is not an exotic journal…

Well, D. Labes offered some help already; my contact is behind the [image]

icon at the upper left hand corner of all my posts.

—
Dif-tor heh smusma 🖖🏼 Довге життя Україна!
Helmut Schütz

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

Alice ☆ 2010-07-19 23:57 (5396 d 17:48 ago) @ Helmut Posting: # 5658 Views: 11,635	Small is beautiful Post reply
	Dear Helmut! Thanks for all your help and your explanation. I had never heard about Biometrika, and I'm wondering why my teacher had never talk about this journal. I think I'll read lots of things there. But now, with all your explanations, all is clear in my head. Thanks a lot for all!!

Helmut
★★★

Vienna, Austria,
2010-07-26 04:01
(5390 d 13:44 ago)

@ Alice
Posting: # 5670
Views: 11,766

Biometrika

Post reply

Dear Alice!

❝ I had never heard about Biometrika, and I'm wondering why my teacher had never talk about this journal.

Well, see here. Founded by by Francis Galton, Karl Pearson, and Walter Weldon. The first issue was published in October 1901 (!!).

—
Dif-tor heh smusma 🖖🏼 Довге життя Україна!
Helmut Schütz

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

d_labes
★★★

Berlin, Germany,
2010-07-15 12:41
(5401 d 05:04 ago)

@ Alice
Posting: # 5632
Views: 11,705

Power to the students

Post reply

Dear Alice!

❝ Maybe have you some reading for help me?

Have you considered to contact the author or the maintainer of "PowerTOST"? He is a nice guy I know. At least sometimes ... :cool:

.

There is a vast amount of literature concerning power and sample size for bioequivalence studies. Most of them must be available in the pharmaceutical company you are training at, if they are professionals.

The most important papers are cited in the Help file of the R package.
If you can't get access to it within the time lines of your training report:
see hint above.

I highly recommend Helmut's lectures accessible here.

—
Regards,

Detlew

Alice
☆

2010-07-15 23:55
(5400 d 17:50 ago)

@ d_labes
Posting: # 5641
Views: 11,615

Power to the students

Post reply

❝ Have you considered to contact the author or the maintainer of "PowerTOST"? He is a nice guy I know. At least sometimes ... :cool: .

Yes, I have considered it... But I was thinking the maintainer was to busy for answer to a little student ;-)

❝ There is a vast amount of literature concerning power and sample size for bioequivalence studies. Most of them must be available in the pharmaceutical company you are training at, if they are professionals.

Yes, I imagine lots of literature exist. I will ask about literature, but I'm not sure I'll find something :-|

.
Maybe I'm too curious when I would to understand Owen's Q function....
If I find nothing about it, I'll think about contact the nice maintainer of "PowerTOST" :-D

!
Maybe he is really a nice guy and maybe he doesn't scare a little student like me ;-)

❝ I highly recommend Helmut's lectures accessible here.

I've already read a part of them. I'll read the other ones soon! There are nice lectures!

Thanks too for your post and for your function (of course) :-)

Hope I'll find the good way of the power !

d_labes
★★★

Berlin, Germany,
2010-07-16 14:56
(5400 d 02:49 ago)

@ Alice
Posting: # 5646
Views: 11,650

Power to the studentinnen

Post reply

Dear Alice,

❝ Maybe I'm too curious when I would to understand Owen's Q function....

IMHO it is always a good habit of a scientist to try to understand in detail whats going on under the hood. And I'm very delighted how engaged and interested a little studentin (german: female sort of) is :ok:

.
Thus keep on snoopy.

I must confess that the documentation of "PowerTOST" is not so very exhaustive, not to say there is nothing regarding the used algorithmns. If I have more spare time I will try to improve.
Meanwhile as a shortcut to the myths of Owen's Q try to :google:

keywords: SAS equivalence test power Owen's Q function.
SAS uses exhaustively Owen's Q function in the power and sample size calculations. And therefore there is some documentation about it and its relation to power calculations.

—
Regards,

Detlew

Alice
☆

2010-07-20 00:17
(5396 d 17:29 ago)

@ d_labes
Posting: # 5659
Views: 11,670

Power to the studentinnen

Post reply

Dear D.Labes!

❝ IMHO it is always a good habit of a scientist to try to understand in detail whats going on under the hood.

I've got a same opinion on it.
And without curiosity, we can't discover lots of things in our world!

❝ I must confess that the documentation of "PowerTOST" is not so very exhaustive, not to say there is nothing regarding the used algorithmns.

You have already done R function and I think it's lots of work. And you take time for answer.
You deliver all the keys!

❝ Meanwhile as a shortcut to the myths of Owen's Q try to :google: keywords: SAS equivalence test power Owen's Q function.

Thanks for this great cleverness! I will snoop there!

Thanks for all your answer and all your help!

d_labes
★★★

Berlin, Germany,
2010-07-26 15:59
(5390 d 01:46 ago)

@ Alice
Posting: # 5678
Views: 11,647

Coming soon ...

Post reply

Dear Alice, dear All!

Since there was a bottleneck on CRAN last two weeks or so, now I'm proud to introduce today:

PowerTOST version 0.6-2

uploaded 2010-07-21, now under checking.

If the new version is available to you (within some days I hope), have a look into the /doc subdirectory. There is now a short "tractatus" (as PDF) about the used mathematical and statistical apparatus. As well as some notes about implementation issues.
Maybe it is useful for some of you ... :cool:

I'm open for proposals to write it in more readable / easier to understand form or to bug reports.

—
Regards,

Detlew