Flaws in evaluation of parallel designs [🇷 for BE/BA]
Hi everybody!
In this post I wrote:
OK, it’s getting even worse!
According to FDA’s Guidancehttp://www.fda.gov/cder/guidance/3616fnl.pdf Statistical Approaches Establishing Bioequivalence (2001), Section VI. Statistical Analysis, B. Data Analysis, 1. Average Bioequivalence, d. Parallel Designs:
In other words, naïve pooling of variances – as given in this post – is not appropriate (at least with the FDA).
Degrees of freedom should be corrected by e.g. the Welch-Satterthwaite-Approximation, and the confidence interval calculated accordingly.
Therefore df=21.431 (M$-Excel rounds down to the nearest integer 21) instead of df=22.
Below is a comparison of results (90% CI) for period 1 of the example data set, using different software packages (general purpose statistics, and ‘spezialized’ packages for PK/BE).
For unequal variances in the manual calculation I used the Satterthwaite approximation, uses the Welch approximation, NCCS: Aspin-Welch, STATISTICA: Milliken-Johnson.
WinNonlin, Kinetica, and EquivTest calculate the CI based on the assumption of equal variances (naïve pooling) only, which:
If assumption(s) are violated, the ‘classical’ t-test becomes liberal (i.e., the CI is too tight; patient’s risk is higher than the nominal 5%). Whereas for equal group sizes this inflation of the α-risk may be small, the more imbalanced a study gets the more liberal the test becomes.
As an example I multiplied AUC-values of subjects 4-6 (test) by three, and removed subjects 22-24 (test). Now we have an imbalanced data set (Ntest: 9, Nreference: 12) with unequal variances (test: 0.564, ref: 0.129; F-ratio test p 0.0272, modified Levene test p 0.107).
Nothing about the methods used is documented in the ‘specialized’ programs’ manuals and online help.
Making things even worse, WinNonlin has an option field for ‘Degrees of Freedom’, where ‘Satterthwaite’ is checked by default. Anyhow, for parallel groups it should be noted, that exactly the same results are obtained as for the other method ‘Residual’!
Wang H and S-C Chow
A practical approach for comparing means of two groups without equal variance assumption
Statist Med 21, 3137–51 (2002)
online resource
Edit: Link corrected for FDA’s new site. [Helmut]
In this post I wrote:
❝ You never know when rounding will hit you - and don't dare asking the software vendor for the algorithm...
OK, it’s getting even worse!
According to FDA’s Guidance
For parallel designs, the confidence interval for the difference of means in the log scale can be computed using the total between-subject variance. As in the analysis for replicated designs (section VI. B.1.b), equal variances should not be assumed.
(my emphasis)In other words, naïve pooling of variances – as given in this post – is not appropriate (at least with the FDA).
Degrees of freedom should be corrected by e.g. the Welch-Satterthwaite-Approximation, and the confidence interval calculated accordingly.
Therefore df=21.431 (M$-Excel rounds down to the nearest integer 21) instead of df=22.
Below is a comparison of results (90% CI) for period 1 of the example data set, using different software packages (general purpose statistics, and ‘spezialized’ packages for PK/BE).
┌────────────────────────┬─────────────────┬────────────────┐
│ Program / Method │ equal var. │ unequal var. │
├────────────────────────┼─────────────────┼────────────────┤
│ ‘manual’ (Excel 2000) │ 63.51 - 110.19% │ 63.4 - 110.25% │
│ R 2.4.1 (2006) │ 63.51 - 110.19% │ 63.4 - 110.22% │
│ NCSS 2001 (2001) │ 63.51 - 110.19% │ 63.4 - 110.22% │
│ STATISTICA 5.1H (1997) │ 63.51 - 110.19% │ 63.4 - 110.22% │
│ WinNonlin 5.2 (2007) │ 63.51 - 110.20% │ not available! │
│ Kinetica 4.4.1 (2007) │ 63.51 - 110.19% │ not available! │
│ EquivTest/PK (2006) │ 63.51 - 110.19% │ not available! │
└────────────────────────┴─────────────────┴────────────────┘
For unequal variances in the manual calculation I used the Satterthwaite approximation, uses the Welch approximation, NCCS: Aspin-Welch, STATISTICA: Milliken-Johnson.
WinNonlin, Kinetica, and EquivTest calculate the CI based on the assumption of equal variances (naïve pooling) only, which:
- may not hold, and
- is against FDA’s guideline.
- almost equal variances are observed in the data set, and
- the data set is balanced.
If assumption(s) are violated, the ‘classical’ t-test becomes liberal (i.e., the CI is too tight; patient’s risk is higher than the nominal 5%). Whereas for equal group sizes this inflation of the α-risk may be small, the more imbalanced a study gets the more liberal the test becomes.
As an example I multiplied AUC-values of subjects 4-6 (test) by three, and removed subjects 22-24 (test). Now we have an imbalanced data set (Ntest: 9, Nreference: 12) with unequal variances (test: 0.564, ref: 0.129; F-ratio test p 0.0272, modified Levene test p 0.107).
equal variances: 81.21% - 190.41%
unequal variances: 76.36% - 202.51%
Nothing about the methods used is documented in the ‘specialized’ programs’ manuals and online help.
Making things even worse, WinNonlin has an option field for ‘Degrees of Freedom’, where ‘Satterthwaite’ is checked by default. Anyhow, for parallel groups it should be noted, that exactly the same results are obtained as for the other method ‘Residual’!
Wang H and S-C Chow
A practical approach for comparing means of two groups without equal variance assumption
Statist Med 21, 3137–51 (2002)
online resource
Edit: Link corrected for FDA’s new site. [Helmut]
—
Dif-tor heh smusma 🖖🏼 Довге життя Україна!
Helmut Schütz
The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
Dif-tor heh smusma 🖖🏼 Довге життя Україна!
Helmut Schütz
The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
Thread locked
Complete thread:
- Flaws in evaluation of parallel designsHelmut 2007-04-17 17:16 [🇷 for BE/BA]
- parallel designs (Welch-Satterthwaite in R) Helmut 2007-04-18 17:05