Bioequivalence and Bioavailability Forum

mittyri ★★ Russia, 2022-06-24 15:43 (843 d 22:03 ago) Posting: # 23079 Views: 3,800	T1E after 1st stage in Power2Stage [Two-Stage / GS Designs] Post reply
	Dear Forum Members, could you please guide me regarding the function to be used for T1E estimation after 1st stage of Potvin methods (Power2Stage R package)? — Kind regards, Mittyri

Helmut
★★★

Vienna, Austria,
2022-06-24 16:33
(843 d 21:13 ago)

@ mittyri
Posting: # 23081
Views: 3,287

T1E after 1st stage in TSDs

Post reply

Hi Mittyri,

❝ could you please guide me regarding the function to be used for T1E estimation after 1st stage of Potvin methods (Power2Stage R package)?

Really in the first stage? Negligible for the combinations where you get the largest TIE overall. Anyhow, your wish is my command:

library(Power2Stage) # alpha0 = 0.05 for method = "C" and alpha = c(0.0294, 0.0294) for method = "B" # and "C" are defaults; GMR = 0.95, targetpower = 0.8, if theta0 = 0.8 or 1.25, # by default nsims = 1e6. CV <- 0.2 n1 <- 12 # That’s an evil combination; commonly the largest # TIE with a relatively small CV and very small n1. res <- data.frame(method = c("B", "C"), alpha0 = 0.05, alpha = 0.0294, TIE.stage1 = NA_real_, TIE.stage2 = NA_real_) for (j in 1:nrow(res)) { tmp <- power.tsd(method = res$method[j], n1 = n1, CV = CV, theta0 = 1.25) res[j, 4:5] <- c(tmp$pBE_s1, tmp$pBE) } res$alpha0[1] <- "" print(res, row.names = FALSE) method alpha0 alpha TIE.stage1 TIE.stage2 B 0.0294 0.028849 0.046273 C 0.05 0.0294 0.035777 0.051100

Belt plus suspenders:

library(PowerTOST) # I don’t recommend that but let n1 = n of a fixed sample design # i.e., high chance of passing in the first stage of a TSD. CV <- 0.2 n1 <- sampleN.TOST(CV = CV, print = FALSE)[["Sample size"]] for (j in 1:nrow(res)) { tmp <- power.tsd(method = res$method[j], n1 = n1, CV = CV, theta0 = 1.25) res[j, 4:5] <- c(tmp$pBE_s1, tmp$pBE) } res$alpha0[1] <- "" print(res, row.names = FALSE) method alpha0 alpha TIE.stage1 TIE.stage2 B 0.0294 0.029142 0.034816 C 0.05 0.0294 0.045318 0.049042

PS: I changed the category from Software. Though you asked for the [image]

-package Power2Stage, perhaps somebody would have the balls to roll out own code. ;-)

—
Dif-tor heh smusma 🖖🏼 Довге життя Україна!
Helmut Schütz

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

mittyri
★★

Russia,
2022-06-24 23:12
(843 d 14:34 ago)

@ Helmut
Posting: # 23084
Views: 3,246

T1E after 1st stage in TSDs

Post reply

Hi Helmut,

❝ tmp <- power.tsd(method = res$method[j], n1 = n1, CV = CV,

❝ theta0 = 1.25)

do I understand it right that if the study is terminated after 1st stage (due to any reason), we are using the same function with n1 (not fixing n2 to 0)

❝

PS: I changed the category from Software. Though you asked for the [image]

-package Power2Stage, perhaps somebody would have the balls to roll out own code. ;-)

I changed the category twice when wrote the post :-D

—
Kind regards,
Mittyri

Helmut
★★★

Vienna, Austria,
2022-06-25 16:07
(842 d 21:39 ago)

@ mittyri
Posting: # 23086
Views: 3,241

T1E after 1st stage in TSDs

Post reply

Hi Mittyri,

[image]

Of course, ElMaestro is right.

There Can Be Only One.

❝ do I understand it right that if the study is terminated after 1st stage (due to any reason), we are using the same function with n1 (not fixing n2 to 0)

Not sure what you mean. For any of the simulated studies the decision scheme of the method is followed. The function returns counters for the studies passing in the 1^st stage (pBE_s1) and of the pooled data in the 2^nd stage (pBE). We can extract all numbers:

library(Power2Stage) # using defaults of method = "B" tmp <- power.tsd(CV = 0.2, n1 = 12, theta0 = 1.25) if (tmp$theta0 == tmp$theta1 | tmp$theta0 == tmp$theta2) { txt <- "Type I Error" } else { txt <- "Power" } cat(formatC(tmp$nsims, digits = 7, big.mark = ","), "simulated studies\nFailed in stage 1 :", formatC(tmp$nsims - tmp$nsims * tmp$pBE_s1 - tmp$pct_s2 / 100 * tmp$nsims, digits = 6, big.mark = ","), sprintf("(%7.4f%%)", 100 * (tmp$nsims - tmp$nsims * tmp$pBE_s1 - tmp$pct_s2 / 100 * tmp$nsims) / tmp$nsims), "\nPassed in stage 1 :", formatC(tmp$pBE_s1 * tmp$nsims, digits = 6, big.mark = ","), sprintf("(%7.4f%%)", 100 * tmp$pBE_s1), "\nProceeded to stage 2 :", formatC(tmp$nsims * tmp$pct_s2 / 100, digits = 6, big.mark = ","), sprintf("(%7.4f%%)", tmp$pct_s2), "\nFailed in stage 2 :", formatC(tmp$nsims * tmp$pct_s2 / 100 - (tmp$pBE -tmp$pBE_s1) * tmp$nsims, digits = 6, big.mark = ","), sprintf("(%7.4f%%)", 100 * (tmp$nsims * tmp$pct_s2 / 100 - (tmp$pBE -tmp$pBE_s1) * tmp$nsims) / tmp$nsims), "\nPassed in stage 2 :", formatC((tmp$pBE -tmp$pBE_s1) * tmp$nsims, digits = 6, big.mark = ","), sprintf("(%7.4f%%)", 100 * (tmp$pBE -tmp$pBE_s1)), "\nFailed in stage 1 or 2:", formatC((1 - tmp$pBE) * tmp$nsims, digits = 6, big.mark = ","), sprintf("(%7.4f%%)", 100 * (1 - tmp$pBE)), "\nPassed in stage 1 or 2:", formatC(tmp$pBE * tmp$nsims, digits = 6, big.mark = ","), sprintf("(%7.4f%%) = empiric", 100 * tmp$pBE), txt, "\n") 1,000,000 simulated studies Failed in stage 1 : 92,550 ( 9.2550%) Passed in stage 1 : 28,849 ( 2.8849%) Proceeded to stage 2 : 878,601 (87.8601%) Failed in stage 2 : 861,177 (86.1177%) Passed in stage 2 : 17,424 ( 1.7424%) Failed in stage 1 or 2: 953,727 (95.3727%) Passed in stage 1 or 2: 46,273 ( 4.6273%) = empiric Type I Error

And:

tmp <- power.tsd(CV = 0.2, n1 = 12, theta0 = 0.95) 100,000 simulated studies Failed in stage 1 : 2,325 ( 2.3250%) Passed in stage 1 : 41,333 (41.3330%) Proceeded to stage 2 : 56,342 (56.3420%) Failed in stage 2 : 13,409 (13.4090%) Passed in stage 2 : 42,933 (42.9330%) Failed in stage 1 or 2: 15,734 (15.7340%) Passed in stage 1 or 2: 84,266 (84.2660%) = empiric Power

❝ ❝

PS: I changed the category from Software. Though you asked for the [image]

-package Power2Stage, perhaps somebody would have the balls to roll out own code. ;-)

❝

❝ I changed the category twice when wrote the post :-D

I see. Feel free to change it to anyone you like.

—
Dif-tor heh smusma 🖖🏼 Довге життя Україна!
Helmut Schütz

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

mittyri
★★

Russia,
2022-06-26 01:29
(842 d 12:17 ago)

@ Helmut
Posting: # 23087
Views: 3,226

Potvin C consequences

Post reply

Hi Helmut (and ElMaestro - sorry to post in this thread, but all your answers are great!),

❝ Of course, ElMaestro is right.

❝

There Can Be Only One.

OK, I understand the logic here.
take a look at some (theoretical) situation: a Sponsor goes with Potvin C (don't ask me why), moreover, he proved that with the current CV and n1 T1E does not exceed 5%.
But as you showed in the lectures and in example above, Potvin C could not ensure 5% for all combinations. Thus, during stage 1 the CV was lower and the resulted T1E was out of 5%.
What would be the right (safe) way out in such situation?

—
Kind regards,
Mittyri

ElMaestro
★★★

Denmark,
2022-06-26 11:11
(842 d 02:35 ago)

@ mittyri
Posting: # 23088
Views: 3,212

Potvin C consequences

Post reply

Hey easy now,

❝ ❝ Of course, ElMaestro is right.

While that sounds good it is only very rarely the truth. :crying:

❝ take a look at some (theoretical) situation: a Sponsor goes with Potvin C (don't ask me why), moreover, he proved that with the current CV and n1 T1E does not exceed 5%.

❝ But as you showed in the lectures and in example above, Potvin C could not ensure 5% for all combinations. Thus, during stage 1 the CV was lower and the resulted T1E was out of 5%.

❝ What would be the right (safe) way out in such situation?

I can't follow you, in particular the parts in red.
Your type I error is not depending on your estimates and thus not depending on whether you went into stage 2 or not. They are depending on the true GMR, the (true) variability, N and the underlying stats assumptions; you may have estimates of the CV and GMR and your N is known.
Note also that when we speak of type I error it is often implied that it is the maximum type I error. By definition, it is on the acceptance borders that you get an idea of the type I errors.

—
Pass or fail!
ElMaestro

mittyri
★★

Russia,
2022-06-27 15:45
(840 d 22:01 ago)

@ ElMaestro
Posting: # 23093
Views: 3,134

Potvin C consequences

Post reply

Hi ElMaestro,

some more questions

❝ Your type I error is not depending on your estimates and thus not depending on whether you went into stage 2 or not.

OK, what about this:

power.tsd(method = "C", CV = 0.2, n1 = 12, theta0 = 1.25)



p(BE)    = 0.0511



power.tsd(method = "C", CV = 0.3, n1 = 12, theta0 = 1.25)



p(BE)    = 0.043743

So T1E (OK, maximum type I error) depends on the CV, isn't it?
Or may be I misunderstood your point and we should not verify T1E during estimation process since we are assuming that the framework was stated as acceptable in the Protocol?

—
Kind regards,
Mittyri

Helmut
★★★

Vienna, Austria,
2022-06-27 16:49
(840 d 20:57 ago)

@ mittyri
Posting: # 23094
Views: 3,124

Potvin C consequences

Post reply

Hi Mittyri,

❝ OK, what about this:

❝ power.tsd(method = "C", CV = 0.2, n1 = 12, theta0 = 1.25)

❝ p(BE) = 0.0511

❝ power.tsd(method = "C", CV = 0.3, n1 = 12, theta0 = 1.25)

❝ p(BE) = 0.043743

❝

❝ So T1E (OK, maximum type I error) depends on the CV, isn't it?

Yes, though not only on the CV but on n₁ as well (though less so).

❝ Or may be I misunderstood your point and we should not verify T1E during estimation process since we are assuming that the framework was stated as acceptable in the Protocol?

In principle yes but I received a few deficiency letters asking for a “post hoc assessement of the Type I Error”. That’s not well-thought-out. Of course, we know n₁ but the CV is just an estimate. The true one (used in the simulations of the framework) is unknown.

There are no problems with “Method B”. In the entire framework of CV/n₁-combinations the T1E is controlled. When you explore a narrow grid (step size of CV 2% and of n₁ 2) you will get this (noncentral t-distribution instead of the shifted central t like in the paper):

[image]
Maximum Type I Error 0.048856 at CV 24% and n₁ 12.

As you already discovered, “Method C” can be problematic – though only in certain rare cases.

[image]
Maximum Type I Error 0.051250 at CV 22% and n₁ 12
(the yellow dot). Thick red contour at the bottom encloses T1E > 0.05.

IMHO, it was unfortunate by Potvin et al. to select the same adjusted α in both methods. Any ‘Type 2’ TSD will require more adjustment than its ‘Type 1’ counterpart.
BTW, 0.0294 in “Method B” is overly conservative; 0.0302 controls the Type I Error as well. On the other hand, for “Method C” you need 0.0282.

Consider to avoid this stuff in the future. See this post and followings for better alternatives.

—
Dif-tor heh smusma 🖖🏼 Довге життя Україна!
Helmut Schütz

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

mittyri
★★

Russia,
2022-06-27 19:13
(840 d 18:33 ago)

@ Helmut
Posting: # 23096
Views: 3,112

from Potvin C to Inverse Normal

Post reply

Dear Helmut,

❝ In principle yes but I received a few deficiency letters asking for a “post hoc assessement of the Type I Error”. That’s not well-thought-out. Of course, we know n₁ but the CV is just an estimate. The true one (used in the simulations of the framework) is unknown.

Fully agree! This thread was started due to deficiency letter my friend showed me some time ago. I thought I missed something in TSD, but now I see a full picture

❝ Consider to avoid this stuff in the future. See this post and followings for better alternatives.

there are zero papers I've seen by Russian experts regarding inverse normal method. I guess this is the reason why the sponsors do not want to have a risk with that invention. "Why should we care if everything works with Potvin?"

—
Kind regards,
Mittyri

Helmut
★★★

Vienna, Austria,
2022-06-27 20:54
(840 d 16:52 ago)

@ mittyri
Posting: # 23098
Views: 3,124

From Potvin C to Inverse Normal: history (and outlook?)

Post reply

Hi Mittyri and all,

❝ ❝ Consider to avoid this stuff in the future. See this post and followings for better alternatives.

❝

❝ there are zero papers I've seen by Russian experts regarding inverse normal method. I guess this is the reason why the sponsors do not want to have a risk with that invention. "Why should we care if everything works with Potvin?"

Obviously it doesn’t work. Maybe it’s [image]

time for a change.

In my studies I used only “Method C” and never received a single (‼) deficiency letter. Why? Not sure. Only because all studies stopped for success in the first stage? Or because I was ‘famous’?¹ Anyhow, later I faced deficiency letters – even for “Method B”.²

Since this was an extremely unsatisfactory situation, I nudged my friends to develop an exact method (i.e., with strict Type I Error control without relying on simulations). Not an easy job, took us two years.^3,4 As an aside, only the 2×2×2 crossover design is covered, but we stated:

[…] the proposed adaptive design would allow to switch from a classical two-period design to a more complex replicate design if it turns out the reference product is highly variable and the within subject-variability has to be determined as well.

Currently science fiction.
Our posters were noticed and Byron Jones started working on the stuff together will a veteran of adaptive designs, Willi Maurer†. In December 2016 the method appeared in a book.⁵ Unfortunately the [image]

code was crude, politely speaking. In February 2018 the paper⁶ – containing the analytical proof and ‘improved’ [image]

code as supplementary information⁷ – was published. The method is extremely flexible when it comes to futility criteria (PE or CI not within certain limits, estimated total sample size above a limit), minimum stage 2 sample size protecting against dropouts, and even sample size re-estimation based on the T/R-ratio observed in the first stage (i.e., fully adaptive). Since stages are evaluated separately, not only an ANOVA can be used but also a mixed-effects model, which is recommended by the FDA and Health Canada. It took Benjamin Lang and Detlew Labes almost half a year to implement the inverse-normal combination method in Power2Stage.⁸ BTW, the method is acceptable for the FDA as well.⁹

I was told that Russian »эксперты« have altercations of the type:

“Helmut said …!” – “No, on the contrary! He said …”

and so on and so forth.
What I really said and keep saying: Use the inverse-normal combination method!

Regrettably a parallel design is another pot of tea. Taking unequal group sizes in the first stage and heterogenicity into account is not trivial. At the very bottom of our todo list.
In the meantime use the simulation-based method¹⁰ available in Power2Stage.

Schütz H. Two-stage designs in bioequivalence trials. Eur J Clin Pharmacol. 2015; 71(3): 271–81. doi:10.1007/s00228-015-1806-2.
Last year I had to deal with a deficiency letter of the Czech agency SÚKL. “Method B” passed with flying colors in the first stage. The agency questioned the reliability of Potvin’s simulations ‘because the grid in the publication’s simulations was too sparse’. Bizarre. Of course, a narrow grid with a step size of 2 for n₁ and 2% for the CV showed no inflated Type I Error as well (maximum 0.048856 for n₁ 12 and CV 24%). Hurray, 1.116 billion simulated studies!
König F, Wolfsegger M, Jaki T, Schütz H, Wassmer G. Adaptive two-stage bioequivalence trials with early stopping and sample size re-estimation. Vienna: 2014; 35^th Annual Conference of the International Society for Clinical Biostatistics. Poster P1.2.88. doi:10.13140/RG.2.1.5190.0967.
König F, Wolfsegger M, Jaki T, Schütz H, Wassmer G. Adaptive two-stage bioequivalence trials with early stopping and sample size re-estimation. Trials. 2015; 16(Suppl 2); P218. doi:10.1186/1745-6215-16-S2-P218.
Patterson SD, Jones B. Bioequivalence and Statistics in Clinical Pharmacology. Boca Raton: CRC Press; 2^nd edition 2017. ISBN 978-1-4665-8520-1. p. 141–187.
Maurer W, Jones B, Chen Y. Controlling the type 1 error rate in two-stage sequential designs when testing for average bioequivalence. Stat Med. 2018; 37(10): 1–21. doi:10.1002/sim.7614.
Maurer W, Jones B, Chen Y. Supplementary Information for ”Controlling the Type I error rate in two-stage sequential adaptive designs when testing for Average Bioequivalence”. Open access.
Labes D, Lang B. Schütz H. Power2Stage: Power and Sample-Size Distribution of 2-Stage Bioequivalence Studies. Package version 0.5-1. 2018-04-03. Current version on CRAN.
Lee J, Feng K, Xu M, Gong X, Sun W, Kim J, Zhang Z, Wang M, Fang L, Zhao L. Applications of Adaptive Designs in Generic Drug Development. Clin Pharm Ther. 2020; 110(1): 32–5. doi:10.1002/cpt.2050.
Fuglsang A. Sequential Bioequivalence Approaches for Parallel Designs. AAPS J. 2014; 16(3): 373–8. doi:10.1208/s12248-014-9571-1. Free Full text.

Seems that this is a hot topic. 210 views in 3½ days. :-D

—
Dif-tor heh smusma 🖖🏼 Довге життя Україна!
Helmut Schütz

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

ElMaestro
★★★

Denmark,
2022-06-27 19:13
(840 d 18:33 ago)

@ mittyri
Posting: # 23095
Views: 3,180

Potvin C consequences

Post reply

Hi mittyri,

❝ So T1E (OK, maximum type I error) depends on the CV, isn't it?

❝ Or may be I misunderstood your point and we should not verify T1E during estimation process since we are assuming that the framework was stated as acceptable in the Protocol?

Depends on the CV, but does not depend on the estimated CV. And that's a very significant difference. You never know anything other than your estimates, i.e. the knowledge that is generated as results of your trial do not determine the (maximum) type I error.

—
Pass or fail!
ElMaestro

mittyri
★★

Russia,
2022-06-27 19:22
(840 d 18:24 ago)

@ ElMaestro
Posting: # 23097
Views: 3,128

true CV ≠ observed CV

Post reply

Hi ElMaestro,

❝ Depends on the CV, but does not depend on the estimated CV. And that's a very significant difference. You never know anything other than your estimates, i.e. the knowledge that is generated as results of your trial do not determine the (maximum) type I error.

you are right, I completely agree. We cannot base TIE estimation on the observed CV.
As Helmut noted, to be on the safe side Potvin C should gone with the wind, so the framework per se should not permit such situations for given prerequisites (or even better to go with Inverse normal)

—
Kind regards,
Mittyri

ElMaestro
★★★

Denmark,
2022-06-24 20:25
(843 d 17:22 ago)

@ mittyri
Posting: # 23082
Views: 3,272

T1E after 1st stage in Power2Stage

Post reply

Hi Mittyri,

❝ could you please guide me regarding the function to be used for T1E estimation after 1st stage of Potvin methods (Power2Stage R package)?

That would be a misnomer, I think. The T1E is a property of the testing approach. There is an alpha at stage 1 and there is an alpha at stage 2 but there is no T1E at stage 1 (or 2). There is just a well-defined T1E for the approach.
Mixing up the terms alpha and T1E is a real danger for 2-stage approaches, since in "normal" trials they are one and the same.

—
Pass or fail!
ElMaestro

mittyri ★★ Russia, 2022-06-24 23:05 (843 d 14:41 ago) @ ElMaestro Posting: # 23083 Views: 3,295	T1E after 1st stage in Power2Stage Post reply
	Hi ElMaestro, ❝ Mixing up the terms alpha and T1E is a real danger for 2-stage approaches, since in "normal" trials they are one and the same. yep, I should reformulate: T1E for the adaptive study finished after 1st stage due to BE goal reached (or some other reason to stop) — Kind regards, Mittyri

ElMaestro
★★★

Denmark,
2022-06-25 00:55
(843 d 12:51 ago)

@ mittyri
Posting: # 23085
Views: 3,217

T1E after 1st stage in Power2Stage

Post reply

Hi mittyri,

❝ yep, I should reformulate: T1E for the adaptive study finished after 1st stage due to BE goal reached (or some other reason to stop)

See my answer above. It does not change regardless of whether you stopped after stage 1, regardless of whether alpha 1 and alpha 2 are the same or different.

A relevant perspective: Imagine you do a normal trial with a 90% CI of 88.88%-99.99%. What is the type I error? The answer is 5%.
That answer -the 5% percent- does not in any way depend on the figures. Same for the two-stage approach. The type 1 error does not change with your data, it doesn't matter if you stop at stage 1 or not.

—
Pass or fail!
ElMaestro

T1E after 1st stage in Power2Stage [Two-Stage / GS Designs]

T1E after 1st stage in TSDs

T1E after 1st stage in TSDs

T1E after 1st stage in TSDs

Potvin C consequences

Potvin C consequences

Potvin C consequences

Potvin C consequences

from Potvin C to Inverse Normal

From Potvin C to Inverse Normal: history (and outlook?)

Potvin C consequences

true CV ≠ observed CV

T1E after 1st stage in Power2Stage

T1E after 1st stage in Power2Stage

T1E after 1st stage in Power2Stage