mittyri
★★  

Russia,
2022-06-24 13:43
(51 d 18:52 ago)

Posting: # 23079
Views: 870
 

 T1E after 1st stage in Power2Stage [Two-Stage / GS Designs]

Dear Forum Members,

could you please guide me regarding the function to be used for T1E estimation after 1st stage of Potvin methods (Power2Stage R package)?

Kind regards,
Mittyri
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2022-06-24 14:33
(51 d 18:02 ago)

@ mittyri
Posting: # 23081
Views: 823
 

 T1E after 1st stage in TSDs

Hi Mittyri,

» could you please guide me regarding the function to be used for T1E estimation after 1st stage of Potvin methods (Power2Stage R package)?

Really in the first stage? Negligible for the combinations where you get the largest TIE overall. Anyhow, your wish is my command:

library(Power2Stage)
# alpha0 = 0.05 for method = "C" and alpha = c(0.0294, 0.0294) for method = "B"
# and "C" are defaults; GMR = 0.95, targetpower = 0.8, if theta0 = 0.8 or 1.25,
# by default nsims = 1e6.

CV  <- 0.2
n1  <- 12
# That’s an evil combination; commonly the largest
# TIE with a relatively small CV and very small n1.

res <- data.frame(method = c("B", "C"), alpha0 = 0.05, alpha = 0.0294,
                  TIE.stage1 = NA_real_, TIE.stage2 = NA_real_)
for (j in 1:nrow(res)) {
  tmp         <- power.tsd(method = res$method[j], n1 = n1, CV = CV,
                           theta0 = 1.25)
  res[j, 4:5] <- c(tmp$pBE_s1, tmp$pBE)
}
res$alpha0[1] <- ""
print(res, row.names = FALSE)

 method alpha0  alpha TIE.stage1 TIE.stage2
      B        0.0294   0.028849   0.046273
      C   0.05 0.0294   0.035777   0.051100


Belt plus suspenders:

library(PowerTOST)
# I don’t recommend that but let n1 = n of a fixed sample design
# i.e., high chance of passing in the first stage of a TSD.

CV  <- 0.2
n1  <- sampleN.TOST(CV = CV, print = FALSE)[["Sample size"]]
for (j in 1:nrow(res)) {
  tmp         <- power.tsd(method = res$method[j], n1 = n1, CV = CV,
                           theta0 = 1.25)
  res[j, 4:5] <- c(tmp$pBE_s1, tmp$pBE)
}
res$alpha0[1] <- ""
print(res, row.names = FALSE)

 method alpha0  alpha TIE.stage1 TIE.stage2
      B        0.0294   0.029142   0.034816
      C   0.05 0.0294   0.045318   0.049042



PS: I changed the category from Software. Though you asked for the [image]-package Power2Stage, perhaps somebody would have the balls to roll out own code. ;-)

Dif-tor heh smusma 🖖 [image]
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
mittyri
★★  

Russia,
2022-06-24 21:12
(51 d 11:23 ago)

@ Helmut
Posting: # 23084
Views: 785
 

 T1E after 1st stage in TSDs

Hi Helmut,

» tmp         <- power.tsd(method = res$method[j], n1 = n1, CV = CV,
»                          theta0 = 1.25)

do I understand it right that if the study is terminated after 1st stage (due to any reason), we are using the same function with n1 (not fixing n2 to 0)

»
PS: I changed the category from Software. Though you asked for the [image]-package Power2Stage, perhaps somebody would have the balls to roll out own code. ;-)

I changed the category twice when wrote the post :-D

Kind regards,
Mittyri
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2022-06-25 14:07
(50 d 18:29 ago)

@ mittyri
Posting: # 23086
Views: 753
 

 T1E after 1st stage in TSDs

Hi Mittyri,

[image]Of course, ElMaestro is right.

There Can Be Only One.


» do I understand it right that if the study is terminated after 1st stage (due to any reason), we are using the same function with n1 (not fixing n2 to 0)

Not sure what you mean. For any of the simulated studies the decision scheme of the method is followed. The function returns counters for the studies passing in the 1st stage (pBE_s1) and of the pooled data in the 2nd stage (pBE). We can extract all numbers:

library(Power2Stage)
# using defaults of method = "B"
tmp <- power.tsd(CV = 0.2, n1 = 12, theta0 = 1.25)
if (tmp$theta0 == tmp$theta1 | tmp$theta0 == tmp$theta2) {
  txt <- "Type I Error"
} else {
  txt <- "Power"
}
cat(formatC(tmp$nsims, digits = 7, big.mark = ","),
    "simulated studies\nFailed in stage 1     :",
    formatC(tmp$nsims - tmp$nsims * tmp$pBE_s1 - tmp$pct_s2 / 100 * tmp$nsims,
            digits = 6, big.mark = ","),
    sprintf("(%7.4f%%)",
            100 * (tmp$nsims - tmp$nsims * tmp$pBE_s1 - tmp$pct_s2 /
            100 * tmp$nsims) / tmp$nsims),
    "\nPassed in stage 1     :",
    formatC(tmp$pBE_s1 * tmp$nsims, digits = 6, big.mark = ","),
    sprintf("(%7.4f%%)", 100 * tmp$pBE_s1),
    "\nProceeded to stage 2  :",
    formatC(tmp$nsims * tmp$pct_s2 / 100, digits = 6, big.mark = ","),
    sprintf("(%7.4f%%)", tmp$pct_s2),
    "\nFailed in stage 2     :",
    formatC(tmp$nsims * tmp$pct_s2 / 100 - (tmp$pBE -tmp$pBE_s1) * tmp$nsims,
            digits = 6, big.mark = ","),
    sprintf("(%7.4f%%)",
    100 * (tmp$nsims * tmp$pct_s2 / 100 - (tmp$pBE -tmp$pBE_s1) * tmp$nsims) / tmp$nsims),
    "\nPassed in stage 2     :",
    formatC((tmp$pBE -tmp$pBE_s1) * tmp$nsims, digits = 6, big.mark = ","),
    sprintf("(%7.4f%%)", 100 * (tmp$pBE -tmp$pBE_s1)),
    "\nFailed in stage 1 or 2:",
    formatC((1 - tmp$pBE) * tmp$nsims, digits = 6, big.mark = ","),
    sprintf("(%7.4f%%)", 100 * (1 - tmp$pBE)),
    "\nPassed in stage 1 or 2:",
    formatC(tmp$pBE * tmp$nsims, digits = 6, big.mark = ","),
    sprintf("(%7.4f%%) = empiric", 100 * tmp$pBE), txt, "\n")


1,000,000 simulated studies
Failed in stage 1     :  92,550 ( 9.2550%)
Passed in stage 1     :  28,849 ( 2.8849%)
Proceeded to stage 2  : 878,601 (87.8601%)
Failed in stage 2     : 861,177 (86.1177%)
Passed in stage 2     :  17,424 ( 1.7424%)
Failed in stage 1 or 2: 953,727 (95.3727%)
Passed in stage 1 or 2:  46,273 ( 4.6273%) = empiric Type I Error


And:

tmp <- power.tsd(CV = 0.2, n1 = 12, theta0 = 0.95)

100,000 simulated studies
Failed in stage 1     :   2,325 ( 2.3250%)
Passed in stage 1     :  41,333 (41.3330%)
Proceeded to stage 2  :  56,342 (56.3420%)
Failed in stage 2     :  13,409 (13.4090%)
Passed in stage 2     :  42,933 (42.9330%)
Failed in stage 1 or 2:  15,734 (15.7340%)
Passed in stage 1 or 2:  84,266 (84.2660%) = empiric Power



» »
PS: I changed the category from Software. Though you asked for the [image]-package Power2Stage, perhaps somebody would have the balls to roll out own code. ;-)
»
» I changed the category twice when wrote the post :-D

I see. Feel free to change it to anyone you like.

Dif-tor heh smusma 🖖 [image]
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
mittyri
★★  

Russia,
2022-06-25 23:29
(50 d 09:06 ago)

@ Helmut
Posting: # 23087
Views: 729
 

 Potvin C consequences

Hi Helmut (and ElMaestro - sorry to post in this thread, but all your answers are great!),

» Of course, ElMaestro is right.
»

There Can Be Only One.


OK, I understand the logic here.
take a look at some (theoretical) situation: a Sponsor goes with Potvin C (don't ask me why), moreover, he proved that with the current CV and n1 T1E does not exceed 5%.
But as you showed in the lectures and in example above, Potvin C could not ensure 5% for all combinations. Thus, during stage 1 the CV was lower and the resulted T1E was out of 5%.
What would be the right (safe) way out in such situation?

Kind regards,
Mittyri
ElMaestro
★★★

Denmark,
2022-06-26 09:11
(49 d 23:24 ago)

@ mittyri
Posting: # 23088
Views: 721
 

 Potvin C consequences

Hey easy now,

» » Of course, ElMaestro is right.

While that sounds good it is only very rarely the truth. :crying:

» take a look at some (theoretical) situation: a Sponsor goes with Potvin C (don't ask me why), moreover, he proved that with the current CV and n1 T1E does not exceed 5%.
» But as you showed in the lectures and in example above, Potvin C could not ensure 5% for all combinations. Thus, during stage 1 the CV was lower and the resulted T1E was out of 5%.
» What would be the right (safe) way out in such situation?

I can't follow you, in particular the parts in red.
Your type I error is not depending on your estimates and thus not depending on whether you went into stage 2 or not. They are depending on the true GMR, the (true) variability, N and the underlying stats assumptions; you may have estimates of the CV and GMR and your N is known.
Note also that when we speak of type I error it is often implied that it is the maximum type I error. By definition, it is on the acceptance borders that you get an idea of the type I errors.

Pass or fail!
ElMaestro
mittyri
★★  

Russia,
2022-06-27 13:45
(48 d 18:51 ago)

@ ElMaestro
Posting: # 23093
Views: 673
 

 Potvin C consequences

Hi ElMaestro,

some more questions

» Your type I error is not depending on your estimates and thus not depending on whether you went into stage 2 or not.

OK, what about this:
power.tsd(method = "C", CV = 0.2, n1 = 12, theta0 = 1.25)

p(BE)    = 0.0511

power.tsd(method = "C", CV = 0.3, n1 = 12, theta0 = 1.25)

p(BE)    = 0.043743


So T1E (OK, maximum type I error) depends on the CV, isn't it?
Or may be I misunderstood your point and we should not verify T1E during estimation process since we are assuming that the framework was stated as acceptable in the Protocol?

Kind regards,
Mittyri
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2022-06-27 14:49
(48 d 17:46 ago)

@ mittyri
Posting: # 23094
Views: 670
 

 Potvin C consequences

Hi Mittyri,

» OK, what about this:
» power.tsd(method = "C", CV = 0.2, n1 = 12, theta0 = 1.25)
» p(BE)    = 0.0511
» power.tsd(method = "C", CV = 0.3, n1 = 12, theta0 = 1.25)
» p(BE)    = 0.043743
»
» So T1E (OK, maximum type I error) depends on the CV, isn't it?

Yes, though not only on the CV but on n1 as well (though less so).

» Or may be I misunderstood your point and we should not verify T1E during estimation process since we are assuming that the framework was stated as acceptable in the Protocol?

In principle yes but I received a few deficiency letters asking for a “post hoc assessement of the Type I Error”. That’s not well-thought-out. Of course, we know n1 but the CV is just an estimate. The true one (used in the simulations of the framework) is unknown.

There are no problems with “Method B”. In the entire framework of CV/n1-combinations the T1E is controlled. When you explore a narrow grid (step size of CV 2% and of n1 2) you will get this (noncentral t-dis­tri­bu­tion instead of the shifted central t like in the paper):

[image]
Maximum Type I Error 0.048856 at CV 24% and n1 12.


As you already discovered, “Method C” can be problematic – though only in certain rare cases.

[image]
Maximum Type I Error 0.051250 at CV 22% and n1 12
(the yellow dot). Thick red contour at the bottom encloses T1E > 0.05.


IMHO, it was unfortunate by Potvin et al. to select the same adjusted α in both methods. Any ‘Type 2’ TSD will require more adjustment than its ‘Type 1’ counterpart.
BTW, 0.0294 in “Method B” is overly conservative; 0.0302 controls the Type I Error as well. On the other hand, for “Method C” you need 0.0282.

Consider to avoid this stuff in the future. See this post and followings for better alternatives.

Dif-tor heh smusma 🖖 [image]
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
mittyri
★★  

Russia,
2022-06-27 17:13
(48 d 15:22 ago)

@ Helmut
Posting: # 23096
Views: 639
 

 from Potvin C to Inverse Normal

Dear Helmut,

» In principle yes but I received a few deficiency letters asking for a “post hoc assessement of the Type I Error”. That’s not well-thought-out. Of course, we know n1 but the CV is just an estimate. The true one (used in the simulations of the framework) is unknown.

Fully agree! This thread was started due to deficiency letter my friend showed me some time ago. I thought I missed something in TSD, but now I see a full picture

» Consider to avoid this stuff in the future. See this post and followings for better alternatives.

there are zero papers I've seen by Russian experts regarding inverse normal method. I guess this is the reason why the sponsors do not want to have a risk with that invention. "Why should we care if everything works with Potvin?"

Kind regards,
Mittyri
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2022-06-27 18:54
(48 d 13:42 ago)

@ mittyri
Posting: # 23098
Views: 638
 

 From Potvin C to Inverse Normal: history (and outlook?)

Hi Mittyri and all,

» » Consider to avoid this stuff in the future. See this post and followings for better alternatives.
»
» there are zero papers I've seen by Russian experts regarding inverse normal method. I guess this is the reason why the sponsors do not want to have a risk with that invention. "Why should we care if everything works with Potvin?"

[image]Obviously it doesn’t work. Maybe it’s [image] time for a change.

In my studies I used only “Method C” and never received a single (‼) defici­ency letter. Why? Not sure. Only because all studies stopped for success in the first stage? Or because I was ‘famous’?1 Anyhow, later I faced defici­ency letters – even for “Method B”.2

Since this was an extremely unsatisfactory situation, I nudged my friends to de­ve­lop an exact method (i.e., with strict Type I Error control without relying on simulations). Not an easy job, took us two years.3,4 As an aside, only the 2×2×2 crossover design is covered, but we stated:

[…] the proposed adaptive design would allow to switch from a classical two-period design to a more complex replicate design if it turns out the reference product is highly vari­able and the within subject-variability has to be determined as well.

Currently science fiction.
Our posters were noticed and Byron Jones started working on the stuff together will a veteran of adap­tive designs, Willi Maurer. In December 2016 the method appeared in a book.5 Un­for­tu­nately the [image] code was crude, politely speaking. In February 2018 the paper6 – containing the analytical proof and ‘improved’ [image] code as supplementary information7 – was published. The method is extre­mely flexible when it comes to futility criteria (PE or CI not within certain limits, estimated total sample size above a limit), minimum stage 2 sample size protecting against drop­outs, and even sample size re-estimation based on the T/R-ratio observed in the first stage (i.e., fully adaptive). Since stages are evaluated separately, not only an ANOVA can be used but also a mixed-effects model, which is recommended by the FDA and Health Canada. It took Ben­ja­min Lang and Det­lew Labes almost half a year to implement the inverse-normal com­bi­nation method in Power2Stage.8 BTW, the method is acceptable for the FDA as well.9

I was told that Russian »эксперты« have altercations of the type:

“Helmut said …!” – “No, on the contrary! He said …”

and so on and so forth.
What I really said and keep saying:  Use the inverse-normal combination method! 

Regrettably a parallel design is another pot of tea. Taking unequal group sizes in the first stage and heterogenicity into account is not trivial. At the very bottom of our todo list.
In the meantime use the simulation-based method10 available in Power2Stage.


  1. Schütz H. Two-stage designs in bioequivalence trials. Eur J Clin Pharmacol. 2015; 71(3): 271–81. doi:10.1007/s00228-015-1806-2.
  2. Last year I had to deal with a deficiency letter of the Czech agency SÚKL. “Method B” pas­sed with flying colors in the first stage. The agency questioned the reliability of Potvin’s si­mu­lations ‘because the grid in the publication’s simulations was too sparse’. Bizarre. Of course, a narrow grid with a step size of 2 for n1 and 2% for the CV showed no inflated Type I Error as well (maxi­mum 0.048856 for n1 12 and CV 24%). Hurray, 1.116 billion simu­lated studies!
  3. König F, Wolfsegger M, Jaki T, Schütz H, Wassmer G. Adaptive two-stage bioequivalence trials with early stopping and sample size re-estimation. Vienna: 2014; 35th Annual Con­fe­rence of the International Society for Cli­ni­cal Biostatistics. Poster P1.2.88. doi:10.13140/RG.2.1.5190.0967.
  4. König F, Wolfsegger M, Jaki T, Schütz H, Wassmer G. Adaptive two-stage bioequivalence trials with early stopping and sample size re-estimation. Trials. 2015; 16(Suppl 2); P218. doi:10.1186/1745-6215-16-S2-P218.
  5. Patterson SD, Jones B. Bioequivalence and Statistics in Clinical Pharmacology. Boca Raton: CRC Press; 2nd edition 2017. ISBN 978-1-4665-8520-1. p. 141–187.
  6. Maurer W, Jones B, Chen Y. Controlling the type 1 error rate in two-stage sequential designs when testing for average bioequivalence. Stat Med. 2018; 37(10): 1–21. doi:10.1002/sim.7614.
  7. Maurer W, Jones B, Chen Y. Supplementary Information for ”Controlling the Type I error rate in two-stage sequential adaptive designs when testing for Average Bioequivalence”. [image] Open access.
  8. Labes D, Lang B. Schütz H. Power2Stage: Power and Sample-Size Distribution of 2-Stage Bio­equi­valence Studies. Package version 0.5-1. 2018-04-03. Current version on CRAN.
  9. Lee J, Feng K, Xu M, Gong X, Sun W, Kim J, Zhang Z, Wang M, Fang L, Zhao L. Appli­ca­tions of Adaptive Designs in Generic Drug Development. Clin Pharm Ther. 2020; 110(1): 32–5. doi:10.1002/cpt.2050.
  10. Fuglsang A. Sequential Bioequivalence Approaches for Parallel Designs. AAPS J. 2014; 16(3): 373–8. doi:10.1208/s12248-014-9571-1. [image] Free Full text.

Seems that this is a hot topic. 210 views in 3½ days. :-D

Dif-tor heh smusma 🖖 [image]
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
ElMaestro
★★★

Denmark,
2022-06-27 17:13
(48 d 15:23 ago)

@ mittyri
Posting: # 23095
Views: 636
 

 Potvin C consequences

Hi mittyri,

» So T1E (OK, maximum type I error) depends on the CV, isn't it?
» Or may be I misunderstood your point and we should not verify T1E during estimation process since we are assuming that the framework was stated as acceptable in the Protocol?

Depends on the CV, but does not depend on the estimated CV. And that's a very significant difference. You never know anything other than your estimates, i.e. the knowledge that is generated as results of your trial do not determine the (maximum) type I error.

Pass or fail!
ElMaestro
mittyri
★★  

Russia,
2022-06-27 17:22
(48 d 15:13 ago)

@ ElMaestro
Posting: # 23097
Views: 635
 

 true CV ≠ observed CV

Hi ElMaestro,

» Depends on the CV, but does not depend on the estimated CV. And that's a very significant difference. You never know anything other than your estimates, i.e. the knowledge that is generated as results of your trial do not determine the (maximum) type I error.

you are right, I completely agree. We cannot base TIE estimation on the observed CV.
As Helmut noted, to be on the safe side Potvin C should gone with the wind, so the framework per se should not permit such situations for given prerequisites (or even better to go with Inverse normal)

Kind regards,
Mittyri
ElMaestro
★★★

Denmark,
2022-06-24 18:25
(51 d 14:11 ago)

@ mittyri
Posting: # 23082
Views: 791
 

 T1E after 1st stage in Power2Stage

Hi Mittyri,

» could you please guide me regarding the function to be used for T1E estimation after 1st stage of Potvin methods (Power2Stage R package)?

That would be a misnomer, I think. The T1E is a property of the testing approach. There is an alpha at stage 1 and there is an alpha at stage 2 but there is no T1E at stage 1 (or 2). There is just a well-defined T1E for the approach.
Mixing up the terms alpha and T1E is a real danger for 2-stage approaches, since in "normal" trials they are one and the same.

Pass or fail!
ElMaestro
mittyri
★★  

Russia,
2022-06-24 21:05
(51 d 11:31 ago)

@ ElMaestro
Posting: # 23083
Views: 775
 

 T1E after 1st stage in Power2Stage

Hi ElMaestro,

» Mixing up the terms alpha and T1E is a real danger for 2-stage approaches, since in "normal" trials they are one and the same.

yep, I should reformulate: T1E for the adaptive study finished after 1st stage due to BE goal reached (or some other reason to stop)

Kind regards,
Mittyri
ElMaestro
★★★

Denmark,
2022-06-24 22:55
(51 d 09:41 ago)

@ mittyri
Posting: # 23085
Views: 766
 

 T1E after 1st stage in Power2Stage

Hi mittyri,

» yep, I should reformulate: T1E for the adaptive study finished after 1st stage due to BE goal reached (or some other reason to stop)

See my answer above. It does not change regardless of whether you stopped after stage 1, regardless of whether alpha 1 and alpha 2 are the same or different.

A relevant perspective: Imagine you do a normal trial with a 90% CI of 88.88%-99.99%. What is the type I error? The answer is 5%.
That answer -the 5% percent- does not in any way depend on the figures. Same for the two-stage approach. The type 1 error does not change with your data, it doesn't matter if you stop at stage 1 or not.

Pass or fail!
ElMaestro
UA Flag
Activity
 Admin contact
22,301 posts in 4,667 threads, 1,585 registered users;
online 2 (0 registered, 2 guests [including 2 identified bots]).
Forum time: Monday 08:36 CEST (Europe/Vienna)

Science… never solves a problem
without creating ten more.    George Bernard Shaw

The Bioequivalence and Bioavailability Forum is hosted by
BEBAC Ing. Helmut Schütz
HTML5