Helmut ★★★ Vienna, Austria, 2020-11-25 11:50 (1449 d 04:50 ago) Posting: # 22084 Views: 11,406 |
|
Dear all, following this post I had a closer look at the GCC-GL (Version 2.4 of March 2016, page 26): 3.1.10 Highly variable drugs or drug products What happens if we pre-specify the widened acceptance range and discover in the study that CVwR ≤30? Assess the study for the conventional AR of 80.00–125.00%? IMHO, that means we have a data-driven decision – which might be false and result in an inflated type I error. I performed simulations acc. to my understanding of the GL: CVwR = CVwT = 30%, balanced 4-period 2-sequence full replicate design (TRTR|RTRT), n = 40 (81% power for θ0 = 0.90), 106 studies with θ0 = 1.25: ~20.6% passed… — Dif-tor heh smusma 🖖🏼 Довге життя Україна! Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes |
d_labes ★★★ Berlin, Germany, 2020-11-25 16:07 (1449 d 00:33 ago) @ Helmut Posting: # 22085 Views: 10,145 |
|
Dear Helmut, ❝ What happens if we pre-specify the widened acceptance range and discover in the study that CVwR ≤30? Assess the study for the conventional AR of 80.00–125.00%? IMHO, that means we have a data-driven decision – which might be false and result in an inflated type I error. Correct. ❝ I performed simulations acc. to my understanding of the GL: ❝ ❝ CVwR = CVwT = 30%, balanced 4-period 2-sequence full replicate design (TRTR|RTRT), n = 40 (81% power for θ0 = 0.90), 106 studies with θ0 = 1.25: ~20.6% passed… Confirmed! At least in magnitude. My result: 20.46% passed. Quick and dirty captured code from power.scABEL.sds. — Regards, Detlew |
Helmut ★★★ Vienna, Austria, 2020-11-26 01:09 (1448 d 15:31 ago) @ d_labes Posting: # 22086 Views: 10,160 |
|
Dear Detlew, ❝ ❝ […] we have a data-driven decision – which might be false and result in an inflated type I error. ❝ ❝ Correct. Shit. ❝ ❝ I performed simulations acc. to my understanding of the GL: […] 106 studies with θ0 = 1.25: ~20.6% passed… ❝ Confirmed! At least in magnitude. ❝ My result: 20.46% passed. ❝ Quick and dirty captured code from power.scABEL.sds. How, did you do that? That’s beyond me. I misused your old subject sim code and added:
PS: Only one of the sim’s failed on the PE restriction. res[(res$lower >= res$L & res$upper <= res$U) & (res$PE <80 | res$PE >125), ]
— Dif-tor heh smusma 🖖🏼 Довге життя Україна! Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes |
d_labes ★★★ Berlin, Germany, 2020-11-26 16:38 (1448 d 00:02 ago) @ Helmut Posting: # 22087 Views: 10,099 |
|
Dear Helmut, ❝ ❝ ❝ […] we have a data-driven decision – which might be false and result in an inflated type I error. ❝ ❝ ❝ ❝ Correct. ❝ ❝ Shit. Why shit? ❝ ❝ ❝ I performed simulations acc. to my understanding of the GL: […] 106 studies with θ0 = 1.25: ~20.6% passed… ❝ ❝ ❝ Confirmed! At least in magnitude. ❝ ❝ My result: 20.46% passed. ❝ ❝ Quick and dirty captured code from power.scABEL.sds. ❝ ❝ How, did you do that? That’s beyond me. Stolen the code of subject data sims and scaled ABE evaluation in the working horse function .pwr.SABE.sds() .Implementing the GCC rules instead of EMA rules or RSABE for FDA is simple. Have a look into the code of the new but undocumented function power.fwl.sds() in the GitHub repository.❝ ...And got 20.568% after eleven (!) hours. Wow! 11 hours for one number! 106 sims in ca. 4-5 sec with my implementation. Some results with setseed=FALSE :power.fwl.sds(CV=0.3, n=40, theta0=0.8, design="2x2x4", setseed=F) 0.2045, 0.2066, 0.2047, 0.2051, 0.2082 Seems we are simulating the same number . — Regards, Detlew |
Helmut ★★★ Vienna, Austria, 2020-11-26 18:14 (1447 d 22:26 ago) @ d_labes Posting: # 22088 Views: 10,225 |
|
Dear Detlew, ❝ ❝ Shit. ❝ ❝ Why shit? Cause it slipped through my attention for almost fifteen years… In South Africa for Cmax fixed limits of 75.00–133.33% may be used (no replicate design needed). ❝ ❝ ❝ Quick and dirty captured code from power.scABEL.sds. ❝ ❝ ❝ ❝ How, did you do that? That’s beyond me. ❝ ❝ Stolen the code of subject data sims and scaled ABE evaluation in the working horse function ❝ Have a look into the code of the new but undocumented function THX! ❝ ❝ ...And got 20.568% after eleven (!) hours. ❝ Wow! 11 hours for one number! 106 sims in ca. 4-5 sec with my implementation. Mühsam ernährt sich das Eichhörnchen. ❝ Some results with ❝ ❝ 0.2045, 0.2066, 0.2047, 0.2051, 0.2082 ❝ Seems we are simulating the same number . Terrible for drugs which are not highly variable (reminds me on the FDA’s RSABE). The design is less important. 2-sequence 4-period (full) replicates — Dif-tor heh smusma 🖖🏼 Довге життя Україна! Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes |
d_labes ★★★ Berlin, Germany, 2020-11-28 12:20 (1446 d 04:20 ago) @ Helmut Posting: # 22091 Views: 9,948 |
|
Dear Helmut, ❝ Stolen the code of subject data sims and scaled ABE evaluation in the working horse function ❝ Have a look into the code of the new but undocumented function there is no need to define and program a new function ! Try the following: # define a new regulator object Result matches the previous calculations. — Regards, Detlew |
Helmut ★★★ Vienna, Austria, 2020-11-29 12:21 (1445 d 04:19 ago) @ d_labes Posting: # 22092 Views: 9,928 |
|
Dear Detlew, ❝ there is no need to define and program a new function ! Great!
PS: Do we have a bug in scABEL() ?
— Dif-tor heh smusma 🖖🏼 Довге життя Україна! Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes |
d_labes ★★★ Berlin, Germany, 2020-11-29 17:22 (1444 d 23:19 ago) (edited on 2020-11-29 17:57) @ Helmut Posting: # 22094 Views: 9,838 |
|
Dear Helmut, ❝ PS: Do we have a bug in ❝ ❝ ❝ ❝ ❝ ❝ ❝ ❝ ❝ ❝ ❝ ❝ Not a bug , it's a feature ! Reason: r_const=0.76. With r_const <- log(1.25)/CV2se(0.3) = 0.7601283... we get CV L U Satisfied? — Regards, Detlew |
Helmut ★★★ Vienna, Austria, 2020-11-30 01:16 (1444 d 15:24 ago) @ d_labes Posting: # 22097 Views: 9,818 |
|
Dear Detlew, ❝ Not a bug , it's a feature ! ❝ Reason: r_const=0.76. ❝ With ❝ ❝ ❝ ❝ ❝ ❝ ❝ ❝ Satisfied? Fuck! I already guessed that. I hate “nice numbers”. — Dif-tor heh smusma 🖖🏼 Довге життя Україна! Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes |
wienui ★ Germany/Oman, 2020-11-29 15:03 (1445 d 01:38 ago) @ Helmut Posting: # 22093 Views: 9,861 |
|
Hi Helmut & Detlew, very interesting. could you please kindly enlighten it more for not great statistician like I, and how could this type I error inflation happen? Thanks in advance — Cheers, Osama |
d_labes ★★★ Berlin, Germany, 2020-11-29 18:51 (1444 d 21:49 ago) @ wienui Posting: # 22095 Views: 9,869 |
|
Dear Osama, ❝ ... how could this type I error inflation happen? Imagine, you have a true CVwR of 30%. That means your true BE limits should be 80% ... 125%. Now you performe a study but obtain an estimate of CVwR of 35%. Given this you use the widened (fixed) BE acceptance range of 75% ... 133.33%. With that acceptance range the BE decision is easier to obtain. This also happens if the true theta0 (GMR) is 125% (or 80%), i.e. bioinequivalence is what we have. Thus more than 5% of BE decision result, that means there is an alpha inflation. Hope this makes sense to you. May be Helmut has a better (paedogogical) explanation because he is an exceptionally gifted preacher . — Regards, Detlew |
Helmut ★★★ Vienna, Austria, 2020-11-30 01:14 (1444 d 15:26 ago) @ d_labes Posting: # 22096 Views: 9,931 |
|
Dear Detlew & Osama, ❝ ❝ ... how could this type I error inflation happen? ❝ May be Helmut has a better (paedogogical) explanation because he is an exceptionally gifted preacher . Only you say so. I’ll try. Let’s start with two power curves (first -script at the end). The 2-sequence 4-period replicate study is designed for ABE with conventional limits, 80% power. Since we are wary, we assume a \(\small{\theta_0}\) (T/R-ratio) of 0.90 and estimate a sample size of 40. One power curve for the conventional limits as planned (──) and one for the widened limits (──). The probability of a type I error (i.e., the consumer risk) to be treated with a formulation which is not BE (true \(\small{\theta_0}\) outside the limits) is ≤5%. You see that the blue power curve intersects 0.05 at \(\small{\theta_0=0.80}\) and \(\small{\theta_0=1.25}\). That means also the chance of falsely passing BE is 5%. The same is applicable to South Africa’s approach where we could use pre-specified widened limits independent from the observed CV. Then we have a TIE of 0.05 as well (the red power curve intersects 0.05 at \(\small{\theta_0=0.75}\) and \(\small{\theta_0=1.\dot{3}}\)). Imagine an even more extreme case than the one Detlew mentioned. We observe in the study a CVwR of 30.01% although the true one is 30%. That means we will use the widened limits of 75.00–133.33% instead of the correct 80.00–125.00%. With the same \(\small{\theta_0=0.80}\) and \(\small{\theta_0=1.25}\) we jump to the red power curve and therefore, have a much higher chance of passing (~40% instead of 5%). Now for the tricky part (sorry): In the previous posts we estimated an inflation of the TIE of ~20%. Why not 40%? Say, the true CVwR is 30% and acc. to the convention the drug is considered to be not highly variable. Hence, we would have to apply the conventional limits. There is a ~50% chance that in the actual study we will observe a CVwR >30% and misclassify the drug/formulation as highly variable and apply the widened limits. But there is also a ~50% chance that we observe a CVwR ≤30% and use the conventional limits. Both together gives the ~20% inflation (actually it is more complicated*). What if we estimate the sample size for the GCC’s approach? With 28 it will be lower (second -script). Since with a CV of 30% we have to use the conventional limits (power at \(\small{\theta_0=0.80}\) and \(\small{\theta_0=1.25}\) will be ~0.16) and we felt into the trap of an inflated type I error. Note that the TIE depends also on the sample size. Hence, it will be smaller than with 40 subjects. If you think about iteratively adjusted α like for the reference-scaling methods (third -script) – examples for the 2x2x4 design (sample sizes estimated for the GCC’s approach):
However, we could face a substantial loss in power (for CV 30% and the adjusted α of ~0.0091 it would drop from 81% to 59%). 1. Power-curves for fixed limits
2. Power curves for sample sizes acc. to the GCC’s approach
3. Iteratively adjusted α for the GCC’s approach
Edit: We implemented regulator = "GCC" in version 1.5-3 (2021-01-18) of PowerTOST . Example:
— Dif-tor heh smusma 🖖🏼 Довге життя Україна! Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes |
wienui ★ Germany/Oman, 2020-11-30 04:30 (1444 d 12:10 ago) @ Helmut Posting: # 22098 Views: 9,812 |
|
Dear Detlew & Helmut, Thank you for the brilliant explanation. ❝ ❝ May be Helmut has a better (paedogogical) explanation because he is an exceptionally gifted preacher . ❝ Only you say so. I’ll try. Kein Wunder, dass Ihr beiden den Titel eines grossen Prediger verdient. — Cheers, Osama |
Helmut ★★★ Vienna, Austria, 2020-11-30 15:42 (1444 d 00:58 ago) @ wienui Posting: # 22099 Views: 9,743 |
|
Dear Osama, ❝ Thank you for the brilliant explanation. You are welcome! In the meantime I added more stuff to my post. If my interpretation of the GL is correct (is it?) and applied as such by members of the GCC, we have a problem if the CVwR observed in the study is ≤30%. What could be done?
— Dif-tor heh smusma 🖖🏼 Довге життя Україна! Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes |
Astea ★★ Russia, 2020-12-02 00:34 (1442 d 16:07 ago) @ Helmut Posting: # 22100 Views: 9,778 |
|
Dear Preachers! You've discovered truly a very interesting feature! But I have some doubts in logical equality of the inflation of TIE and consumer's risk. Can you please explain my faults in the following reasoning? Suppose we expect drug A to be highly variable (in the previous study somewhere in Antarctica W. Oodendijk et al. have got CV>30% for the reference drug). Which of the following options should we prefer to write in the protocol in order to care of the customer: a). Use pre-specified wider limits 75-133 for Cmax (no inflation?) b). Use the GCC-GL approach (inflation up to 21%?) Suppose that at the end of the trial we get CV≤30% and CI within 75-133, but out of 80-125. Then for the a-approach we should conclude the drug BE, for the b-approach - fail to conclude BE. That is the risk of the customer to get a bad product is higher in the first approach if we define "a bad product" as a non-HVD with the limits out of 80-125. The difference is in the fact that in the first approach we proclaim the drug to be good if it is within the limits 75-133. Until about 2013 there were a lot of studies in Russia with 75-133 limits for Cmax even for non-HVD drugs. — "Being in minority, even a minority of one, did not make you mad" |
Helmut ★★★ Vienna, Austria, 2020-12-02 02:37 (1442 d 14:04 ago) @ Astea Posting: # 22101 Views: 9,521 |
|
Hi Nastia ❝ […] I have some doubts in logical equality of the inflation of TIE and consumer's risk. Can you please explain my faults in the following reasoning? ❝ ❝ Suppose we expect drug A to be highly variable (in the previous study somewhere in Antarctica W. Oodendijk et al. have got CV>30% for the reference drug). The problem starts already here. How reliable is Oodendijk’s result? Is it the only one? Did the agency agree that the drug is HV and wider limits can be used? ❝ Which of the following options should we prefer to write in the protocol in order to care of the customer: ❝ ❝ a). Use pre-specified wider limits 75-133 for Cmax (no inflation?) ❝ b). Use the GCC-GL approach (inflation up to 21%?) The crucial point is what we consider a “clinically not relevant \(\small{\Delta}\)”. Muse a bit on these goodies: $$\small{\Delta=20\%\implies\left\{\theta_1=80.00\%,\,\theta_2=125.00\%\right\}}\tag{1}$$ $$\small{\Delta=25\%\implies\left\{\theta_1=75.00\%,\,\theta_2=133.3\dot{3}\%\right\}}\tag{a}$$ $$\small{\Delta\: \overset{{\color{Red} ?}}{\rightarrow}\,\begin{vmatrix} \widehat{CV_\textrm{wR}}\leq30\%\rightarrow \widehat{\Delta}=20\%\\ \widehat{CV_\textrm{wR}}>30\%\rightarrow \widehat{\Delta}=25\% \end{vmatrix}\implies\begin{Bmatrix} \theta_1=80.00\%,\,\theta_2=125.00\%\\ \theta_1=75.00\%,\,\theta_2=133.3\dot{3}\% \end{Bmatrix}}\tag{b}$$ \(\small{(1)}\) and \(\small{(\textrm{a})}\) are straightforward. Fixed limits, type I error always ≤ the nominal \(\small{\alpha}\). \(\small{(\textrm{b})}\) is data-driven (like ABEL and RSABE), since it depends on the estimated \(\small{CV_\textrm{wR}}\). The Null-hypothesis is like Schrödinger’s cat – or Wigner’s friend, if you prefer. The study (not based on clinical grounds by the applicant and regulator like in \(\small{(1)}\) and \(\small{(\textrm{a})}\)) “decides” which \(\small{\widehat{\Delta}}\) is acceptable for the patient. That’s not a particularly good idea. By definition (‼) any framework (or a pre-test) might lead to a false decision and hence, inflates the TIE. That’s a multiplicity problem, which – if not adjusted – will increase the familywise error rate. ❝ Suppose that at the end of the trial we get CV≤30% and CI within 75-133, but out of 80-125. ❝ Then for the a-approach we should conclude the drug BE, … If (a) was stated in the protocol and accepted by the agency, fine. The CV is interesting though not relevant. Try the function CVCL() in PowerTOST . Might be pure chance (well include >30%).❝ … for the b-approach - fail to conclude BE. No risk, no fun. ❝ That is the risk of the customer to get a bad product is higher in the first approach if we define "a bad product" as a non-HVD with the limits out of 80-125. Nope. In (a) you accept beforehand that \(\small{\Delta=25\%}\) is not relevant for the patient. But again: You don’t assess the CV at all. Maybe it is HV indeed (like in Antarctica). ❝ The difference is in the fact that in the first approach we proclaim the drug to be good if it is within the limits 75-133. Correct. ❝ Until about 2013 there were a lot of studies in Russia with 75-133 limits for Cmax even for non-HVD drugs. Interesting. However:
— Dif-tor heh smusma 🖖🏼 Довге життя Україна! Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes |
Astea ★★ Russia, 2020-12-02 10:46 (1442 d 05:54 ago) @ Helmut Posting: # 22102 Views: 9,447 |
|
Dear Helmut! Thank you for the prompt answer! ❝ The problem starts already here. How reliable is Oodendijk’s result? Is it the only one? The reliability of someone else's data - that is the question (especially when one of the authors says "waouf" ❝ The crucial point is what we consider a “clinically not relevant \(\small{\Delta}\)” As far as we (and the Agency) proclaim 25% to be clinically not relevant there is no difference in the rate of the harm for the customer's health independently from the a- or b- approach. For the b-approach he'll just receive not worser drug or doesn't receive it at all. ❝ Try the function I try:
As CI is shifted to the right - does it mean that for these initial conditions the probability of the conclusion of HV is higher? (By the way shouldn't we lower the degrees of freedom for the CV of the reference drug? 3*40-3 should correspond to the common CV of the Test and Reference, shouldn't it?) — "Being in minority, even a minority of one, did not make you mad" |
Helmut ★★★ Vienna, Austria, 2020-12-02 12:34 (1442 d 04:06 ago) @ Astea Posting: # 22103 Views: 9,420 |
|
Hi Nastia, ❝ ❝ The problem starts already here. How reliable is Oodendijk’s result? Is it the only one? ❝ ❝ The reliability of someone else's data - that is the question (especially when one of the authors says "waouf" Willard Oodendijk twittered and Nemo Macron said “waouf”. ❝ ❝ The crucial point is what we consider a “clinically not relevant \(\small{\Delta}\)” ❝ ❝ As far as we (and the Agency) proclaim 25% to be clinically not relevant there is no difference in the rate of the harm for the customer's health independently from the a- or b- approach. For the b-approach he'll just receive not worser drug or doesn't receive it at all. Here you err. In (a) all is good. In (b) everything is in a flux; the applicant and agency agree only that the acceptable risk may be either 20% or 25%. We are dealing with average BE. Classifying HVD(P)s based on CVwR is fine in principle. However, once we make this classification post hoc (based on \(\small{\widehat{CV_\textrm{wR}}}\)), troubles start. Hence, I don’t like* the reference-scaling methods and (b) as well. ❝ ❝ Try the function ❝ I try: ❝
❝ ❝ ❝ ❝ As CI is shifted to the right … Skewed to the right because the variance follows a \(\small{\chi^2}\)-distribution. ❝ … does it mean that for these initial conditions the probability of the conclusion of HV is higher? Yes (for any condition). ❝ (By the way shouldn't we lower the degrees of freedom for the CV of the reference drug? 3*40-3 should correspond to the common CV of the Test and Reference, shouldn't it?) Oops, one more degree of freedom! In the 2-sequence 4-period replicate design we have df = 3n – 4 for the pooled CVw. Following the EMA’s model for the estimation of CVwR we have one factor (the treatment) less in the model and therefore, df = 3n – 3:
— Dif-tor heh smusma 🖖🏼 Довге життя Україна! Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes |
Astea ★★ Russia, 2020-12-02 16:28 (1442 d 00:12 ago) @ Helmut Posting: # 22104 Views: 9,359 |
|
Dear Helmut! Thank you for the explanation! ❝ Here you err. In (a) all is good. In (b) everything is in a flux; the applicant and agency agree only that the acceptable risk may be either 20% or 25%. I just meant to say that the acceptable risk of either 20% or 25% is anyway less or equal to 25%. ❝ Oops, one more degree of freedom! In the 2-sequence 4-period replicate design we have df = 3n – 4 for the pooled CVw. Following the EMA’s model for the estimation of CVwR we have one factor (the treatment) less in the model and therefore, df = 3n – 3: How does this df correspond to the residual df of ANOVA for getting CVWR? I thought that there should be only 40-2=38 degrees of freedom - because from the point of view of the reference drug the full replicate turns to standart 2-way, is it right? — "Being in minority, even a minority of one, did not make you mad" |
Helmut ★★★ Vienna, Austria, 2020-12-02 17:18 (1441 d 23:22 ago) @ Astea Posting: # 22105 Views: 9,325 |
|
Hi Nastia, ❝ How does this df correspond to the residual df of ANOVA for getting CVWR? I thought that there should be only ❝ 40-2=38 degrees of freedom - because from the point of view of the reference drug the full replicate turns to standart 2-way, is it right? I stand corrected!
— Dif-tor heh smusma 🖖🏼 Довге життя Україна! Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes |
d_labes ★★★ Berlin, Germany, 2020-12-02 20:30 (1441 d 20:10 ago) @ Helmut Posting: # 22106 Views: 9,304 |
|
Deatr Both, ❝ Willard Oodendijk twittered and Nemo Macron said “waouf”. forgive me old fart, but whois Mr. Oodendijk? Shold I know him? If yes, why? Enlighten me, please. Maybe I could learn something new in my old age. — Regards, Detlew |
Astea ★★ Russia, 2020-12-02 20:58 (1441 d 19:42 ago) @ d_labes Posting: # 22107 Views: 9,303 |
|
Dear Detlew! ❝ forgive me old fart, but whois Mr. Oodendijk? Shold I know him? If yes, why? Enlighten me, please. That guy is from the neighbouring thread — "Being in minority, even a minority of one, did not make you mad" |
Helmut ★★★ Vienna, Austria, 2020-12-23 13:18 (1421 d 03:22 ago) @ wienui Posting: # 22157 Views: 9,219 |
|
Dear Osama and neds, I updated the development version 1.5.2.9000 of PowerTOST on GitHub (it’s not on CRAN yet). If you want to give it a try:
Some examples in the following. Business as usual (ABE):
Note that we are close to the switching CVwR 30%. What about ABEL?
Using the new argument regulator = "GCC" .
Iteratively adjust α:
Increase the sample size to maintain power (show progress of iterations):
Inspect the plots of this post again. If the true CVwR > 30% it might misclassified as well but this time towards the conventional limits and the TIE is not inflated. Hence, we get the same sample size by
Inflation of the Type I error in different approaches (2-sequence, 4-period full replicate designs):
Edit 2021-01-18: PowerTOST 1.5-3 on CRAN. — Dif-tor heh smusma 🖖🏼 Довге життя Україна! Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes |