Bioequivalence and Bioavailability Forum

Dear all,

I was asked whether it is possible to validate PhEq_bootstrap (see this post). Sure. Since the source-code is available and if one is familiar with Object Pascal even a “white-box” validation is doable. 
I had a quick look based on the example data sets given by Shah et al.* (which is referred in the software’s documentation).

I got for the 90% CI (PhEq_bootstrap v1.2, 2014-06-22, 64bit Windows):

        500 Bootstraps              1,000 Bootstraps
# Shah et al.   PhEq_bootstrap  Shah et al.   PhEq_bootstrap
1 52.79, 68.15   53.10, 67.65   53.01, 68.34   52.82, 67.85
2 48.33, 53.68   48.01, 53.50   48.25, 53.69   48.16, 53.64
3 48.56, 54.10   48.40, 54.49   48.54, 54.56   48.12, 54.14
4 48.39, 51.55   48.23, 51.45   48.38, 51.59   48.31, 51.51
5 46.11, 50.09   45.90, 49.69   46.05, 50.04   46.00, 50.00

Good news. With one exception (test batch 1, 500 bootstraps) the lower CL is more conservative than the ones reported by Vinod. With 1,000 bootstraps results are always slightly more conservative.

Now for the bad news. There is one thing stupid in the software when it comes to validation (line 535 of the source Conv_to_learn1.pas):

RandSeed:=random(random(1000000))+random(random(1000000));

That means that the seed of the (pseudo)random number generator is always a random number itself. Therefore, it is impossible to reproduce runs. Try it: you will not get exactly my results.
IMHO, that’s bad coding practice. In our packages (PowerTOST, Power2Stage) we have an option to work either with a fixed or a random seed. With the former (which is the default) it is possible to reproduce a given run. Detlew’s randomizeBE is even more flexible. You can work with a random seed and the output gives the seed used. Then you can use this seed as an argument to reproduce the run. Clever.

OK, how reproducible are runs of PhEq_bootstrap?

run      5,000 Bootstraps    25,000 Bootstraps
         ƒ2*     90% CI       ƒ2*     90% CI
 1     59.793 52.77, 67.66  59.839 52.81, 67.72
 2     59.786 52.75, 67.70  59.840 52.81, 67.72
 3     59.864 52.82, 67.71  59.839 52.81, 67.72
 4     59.829 52.77, 67.75  59.840 52.81, 67.72
 5     59.840 52.81, 67.70  59.839 52.81, 67.72
 6     59.850 52.82, 67.73  59.843 52.81, 67.72
 7     59.823 52.77, 67.71  59.842 52.81, 67.72
 8     59.843 52.84, 67.70  59.839 52.81, 67.72
 9     59.859 52.77, 67.75  59.836 52.81, 67.72
10     59.803 52.75, 67.70  59.841 52.81, 67.72
11     59.864 52.82, 67.75  59.836 52.81, 67.72
12     59.805 52.75, 67.71  59.840 52.81, 67.72
13     59.860 52.84, 67.72  59.836 52.81, 67.72
14     59.843 52.77, 67.73  59.839 52.81, 67.72
15     59.808 52.77, 67.66  59.840 52.81, 67.72
min    59.786 52.75, 67.66  59.836 52.81, 67.72
Q I    59.806 52.77, 67.70  59.839 52.81, 67.72
med    59.840 52.77, 67.71  59.839 52.81, 67.72
Q III  59.855 52.82, 67.73  59.840 52.81, 67.72
max    59.864 52.84, 67.75  59.843 52.81, 67.72
CV (%) 0.0449 0.063  0.041  0.0034 0.000  0.000

Not so nice. With 5,000 bootstraps the range of the lower CL is 0.07.
Imagine that you claimed similarity (ƒ₂ 50) and an assessor asks you to demonstrate how you obtained it. You fire up the software with its default of 5,000 bootstraps and are slapped in the face with:

f2* = 49.93  Similarity NOT confirmed: Lower CI is below the limit (f2 = 50)

Oops!
Bootstrapping converges to the true value with increasing repetitions. If you want to get reproducible results you have to use a workaround and go for 25,000+ bootstraps. Then the results are stable but you have to accept ~80MB result-files…
Of course, this is only valid for these data sets (12 units, 4 sampling time points). Good luck with more time points.

---

• Shah VP, Tsong Y, Sathe P, Liu J-P. In Vitro Dissolution Profile Comparison—Statistics and Analysis of the Similarity Factor, f₂. Pharm Res. 1998;15(6):889–96. doi:10.1023/A:1011976615750.