Log in
RegisterAdvantages – performance measure [Two-Stage / GS Designs]
posted by Helmut 
– Vienna, Austria, 2015-06-05 15:29 (3579 d 01:50 ago) – Posting: # 14918
Views: 19,691
Dear both!
Tricky. Essentially they compare adaptive designs in terms of the expected total sample and power to a fixed sample design. They call the latter “ideal” which is only true for a known CV. Such a comparison is not fair but the only one we probably have.
Below some stuff (the adjusted HP is 0.0413, not 0.416 – which is for OF). “Type 1”, T/R 0.95, target power 0.80. N is the sample size of the “ideal” design or the expected average sample size in the TSDs. f1 is the ratio of sample sizes and f2 the ratio of expected power:
How to calculate their APS (average performance score) combining the sample size and power is beyond me.
I think there is no general rule (i.e., independent from the expected CV and planned stage 1 sample size) to judge which method performs “best”. For low CVs the larger average sample size may be outweighed by higher power (leaving some “headroom” for the ratio). If the CV is higher than ~0.25 any TSD with such a small n1 will lack power. Let’s play the game with an expected CV of 0.3 and an n1 of 24:
The winner is the symmetric split for my personal favorite n1 ~0.75 of fixed.
Furthermore, in TSDs the distribution of expected total sample sizes is not necessarily normal. Actually it might be even bimodal (with increasing α1 the distribution gets “contaminated” by the fraction of studies stopping in the first stage). 105 sim’s each:
![[image]](img/uploaded/image311.png)
Whether it makes sense to compare designs based on the arithmetic mean (or even the median) remains an open issue.
❝ Liu et al.
❝ "Evaluating the adaptive performance of flexible sample size designs with treatment difference in an interval"
❝ Stat Med 2008; 27:584–596.
❝
❝ But don't ask me for details!
Tricky. Essentially they compare adaptive designs in terms of the expected total sample and power to a fixed sample design. They call the latter “ideal” which is only true for a known CV. Such a comparison is not fair but the only one we probably have.
Below some stuff (the adjusted HP is 0.0413, not 0.416 – which is for OF). “Type 1”, T/R 0.95, target power 0.80. N is the sample size of the “ideal” design or the expected average sample size in the TSDs. f1 is the ratio of sample sizes and f2 the ratio of expected power:
CV method alpha(s) N power f1 f2
0.15 fixed 0.05 – 12 0.8305 – –
adjPotvin 0.0302 0.0302 13.5 0.8830 1.123 1.063
adjHP 0.001 0.0413 14.2 0.8825 1.187 1.063
adjOF 0.005 0.0416 14.1 0.8738 1.179 1.052
0.25 fixed 0.05 – 28 0.8074 – –
adjPotvin 0.0302 0.0302 32.0 0.8126 1.143 1.006
adjHP 0.001 0.0413 30.5 0.7967 1.090 0.987
adjOF 0.005 0.0416 30.4 0.7966 1.087 0.987
0.40 fixed 0.05 – 66 0.8053 – –
adjPotvin 0.0302 0.0302 78.2 0.7500 1.184 0.931
adjHP 0.001 0.0413 70.7 0.7516 1.071 0.933
adjOF 0.005 0.0416 70.5 0.7528 1.069 0.935
0.70 fixed 0.05 – 174 0.8031 – –
adjPotvin 0.0302 0.0302 205.6 0.7252 1.181 0.903
adjHP 0.001 0.0413 185.6 0.7296 1.067 0.908
adjOF 0.005 0.0416 185.1 0.7293 1.064 0.908How to calculate their APS (average performance score) combining the sample size and power is beyond me.
I think there is no general rule (i.e., independent from the expected CV and planned stage 1 sample size) to judge which method performs “best”. For low CVs the larger average sample size may be outweighed by higher power (leaving some “headroom” for the ratio). If the CV is higher than ~0.25 any TSD with such a small n1 will lack power. Let’s play the game with an expected CV of 0.3 and an n1 of 24:
method alpha(s) N power f1 f2
fixed 0.05 – 40 0.8158 – –
adjPotvin 0.0302 0.0302 39.4 0.8284 0.984 1.015
adjHP 0.001 0.0413 42.3 0.8110 1.057 0.994
adjOF 0.005 0.0416 41.9 0.8104 1.048 0.993The winner is the symmetric split for my personal favorite n1 ~0.75 of fixed.
Furthermore, in TSDs the distribution of expected total sample sizes is not necessarily normal. Actually it might be even bimodal (with increasing α1 the distribution gets “contaminated” by the fraction of studies stopping in the first stage). 105 sim’s each:
Whether it makes sense to compare designs based on the arithmetic mean (or even the median) remains an open issue.
—
Dif-tor heh smusma 🖖🏼 Довге життя Україна!![[image]](https://static.bebac.at/pics/Blue_and_yellow_ribbon_UA.png)
Helmut Schütz
![[image]](https://static.bebac.at/img/CC by.png)
The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
Dif-tor heh smusma 🖖🏼 Довге життя Україна!
Helmut Schütz
The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
Complete thread:
- Does unequal alpha distribution make sense? Dr_Dan 2015-05-19 08:29 [Two-Stage / GS Designs]
![[Mix]](https://static.bebac.at/img/mix.png "open in Mix view")
- Maybe; but watch the TIE! Helmut 2015-05-28 15:04
- Advantages d_labes 2015-06-05 09:35
- Advantages ElMaestro 2015-06-05 10:07
- Advantages – performance measure d_labes 2015-06-05 11:21
- Advantages – performance measureHelmut 2015-06-05 13:29
- Advantages – performance measure d_labes 2015-06-05 21:27
- Advantages – performance measure Helmut 2015-06-05 23:29
- Histogram wonder d_labes 2015-06-06 12:58
- Histogram charlatanry Helmut 2015-06-09 13:43
- Histogram beauty d_labes 2015-06-09 14:55
- Histogram beast Helmut 2015-06-09 15:24
- Histogram beauty d_labes 2015-06-09 14:55
- Histogram charlatanry Helmut 2015-06-09 13:43
- Histogram wonder d_labes 2015-06-06 12:58
- Advantages – performance measure Helmut 2015-06-05 23:29
- Advantages – performance measure d_labes 2015-06-05 21:27
- Advantages – performance measureHelmut 2015-06-05 13:29
- Advantages – performance measure d_labes 2015-06-05 11:21
- Advantages ElMaestro 2015-06-05 10:07
- Advantages d_labes 2015-06-05 09:35
- Does unequal alpha distribution make sense? Dr_Dan 2015-05-29 08:33
- Does unequal alpha distribution make sense? ElMaestro 2015-05-29 09:31
- Does unequal alpha distribution make sense? nobody 2015-05-29 12:56
- Old beliefs die hard Helmut 2015-05-29 13:27
- Old beliefs die hard nobody 2015-05-29 13:41
- Old beliefs die hard Helmut 2015-05-29 18:08
- Old beliefs die hard ElMaestro 2015-05-29 18:57
- Old beliefs die hard nobody 2015-05-29 20:06
- ♩ ♪♫♬`·.¸¸.·´`·.¸¸. Helmut 2015-06-02 01:52
- TSD only an option in exceptional cases Dr_Dan 2015-06-02 08:00
- TSD only an option in exceptional cases nobody 2015-06-02 09:23
- TSD only an option in exceptional cases Dr_Dan 2015-06-02 08:00
- Old beliefs die hard ElMaestro 2015-05-29 18:57
- Old beliefs die hard Helmut 2015-05-29 18:08
- Old beliefs die hard nobody 2015-05-29 13:41
- Old beliefs die hard Helmut 2015-05-29 13:27
- Does unequal alpha distribution make sense? nobody 2015-05-29 12:56
- Does unequal alpha distribution make sense? ElMaestro 2015-05-29 09:31
- Banana splits Helmut 2015-06-02 00:48
- Maybe; but watch the TIE! Helmut 2015-05-28 15:04
Ing. Helmut Schütz