Helmut
★★★
avatar
Homepage
Vienna, Austria,
2020-08-13 14:22
(111 d 06:48 ago)

Posting: # 21868
Views: 2,861
 

 Partial replicate design: reference(s)? [RSABE / ABEL]

Dear all,

I try to figure out when – and hopefully why – the partial replicate design TRR|RTR|RRT entered the scene.

I must confess that I never heard about it till I was asked at a workshop in Ahmedabad in 2008. I knew only another partial replicate, the so-called extra-reference design TRR|RTR 1 (which should be avoided since it is biased in the presence of period effects).
The first paper 2 of the ‘Two Lászlós’ dealt solely with the two-sequence, three-period full replicate design TRT|RTR. It laid the foundations of reference-scaled ABE.
The FDA’s guidance of 2001 stated in Appendix B.2.:

… the two-sequence, three-period design TRR|RTT is thought
to be optimal among three-period replicated crossover designs.

Fine with me. Same design characteristics, balanced like the TRT|RTR. Not a single word about the partial replicate.

The partial replicate is given in the FDA’s progesterone guidance of April 2010 and by the EMA in Annex I of September 2016 (appeared first in Rev. 3 of the Q&A document in January 2011) though regulatory documents  scientific justification.

If you know any publication (preferrably prior to 2010), please let me know.


  1. Chen KW, Chow SC, Li G. A Note on Sample Size Determination for Bioequivalence Studies with Higher-order Crossover Designs. J Pharmacokin Biopharm. 1997;25(6):753–65. doi:10.1023/a:1025738019069.
  2. Endrényi L, Tóthfalusi L. Regulatory Conditions for the Determination of Bioequivalence of Highly Variable Drugs. J. Pharm Pharmaceut Sci. 2009;12(1):138–49. [image] Open access.

Dif-tor heh smusma 🖖
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
ElMaestro
★★★

Belgium?,
2020-08-13 15:23
(111 d 05:47 ago)

(edited by ElMaestro on 2020-08-13 15:43)
@ Helmut
Posting: # 21869
Views: 2,493
 

 Partial replicate design: reference(s)?

Hi Hötzi,

it is very interesting. I looked at Chow&Liu's 2009 version of the holy scriptures and they also seem not to mention RTR/RRT/TRR.

I can easily imagine that some innovator had a presub meeting with FDA (about which not much is in the public domain) and then the idea caught on in a rather un-public way and later got the nod from EMA. It is speculation of course but such things would often happen at the initiative in the private sector behind the curtains.
I don't recall the details of dossiers I assessed, I am fairly sure that I (incompetently, of course) assessed BE trials based on RTR/RRT/TRR designs from 2005 and onwards, but I am not 100% sure. Had I not sniffed all that glue, memories of the past would likely have been less foggy.

It might also be that one of the consortia like PQRI could have looked into the matter and released a white paper. I have no idea, but what a mystery :-)
My money is on Walter Hauck, if you can get hold of him, he will tell you where it all came from.


Update: Kamal Midha referred to it as ABE3 in a presentation from 2006.
https://wayback.archive-it.org/7993/20170405065959/https://www.fda.gov/ohrms/dockets/ac/06/slides/2006-4241s2_3_files/frame.htm

Is there an easy way to download that thing, by the way????

I could be wrong, but...

Best regards,
ElMaestro

No, of course you do not need to audit your CRO if it was inspected in 1968 by the agency of Crabongostan.
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2020-08-13 17:03
(111 d 04:07 ago)

@ ElMaestro
Posting: # 21871
Views: 2,460
 

 Partial replicate design: reference(s)?

Hi ElMaestro,

» Update: Kamal Midha referred to it as ABE3 in a presentation from 2006.

Will be difficult to ask him since he retired and moved to Bermuda…

» Is there an easy way to download that thing, by the way????

Your wish is my command.
[image] Internet Archive October 2017.

Dif-tor heh smusma 🖖
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
mittyri
★★  

Russia,
2020-08-13 16:21
(111 d 04:49 ago)

@ Helmut
Posting: # 21870
Views: 2,467
 

 Hyslop's Alternative Cross-over Designs for Individual Bioequivalence

Dear Helmut,

» If you know any publication (preferrably prior to 2010), please let me know.

here ya go

Kind regards,
Mittyri
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2020-08-13 17:08
(111 d 04:02 ago)

@ mittyri
Posting: # 21872
Views: 2,459
 

 Terry’s homebrew

Hi mittyri,

» here ya go

Oh pleeeze, not that one!
Of course, I have it but forgot that it deals with the partial replicate. I didn’t look at it for years cause you need a magnifying glass to read the tables (where the LaTeX screwed up making the formulas unusable).
BTW, it was never published in a peer-reviewed journal… IMHO, proceedings are just one little step above “personal communication”.

Dif-tor heh smusma 🖖
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
mittyri
★★  

Russia,
2020-08-13 18:09
(111 d 03:01 ago)

@ Helmut
Posting: # 21873
Views: 2,435
 

 impressive homebrew

Hi Helmut,

» BTW, it was never published in a peer-reviewed journal… IMHO, proceedings are just one little step above “personal communication”.

Of course you're right. But it looks like it was so impressive that Dr.Patterson cited it in his dissertation.
I did not find any other Hyslop's publications mentioning that design.
Sometimes proceedings are stronger than many articles when you are honored person chosen by FDA;-)

Kind regards,
Mittyri
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2020-08-14 13:21
(110 d 07:49 ago)

@ mittyri
Posting: # 21876
Views: 2,382
 

 impressive indeed

Hi mittyri,

» […] Dr.Patterson cited it in his dissertation.

OK, a Ph.D. thesis (even when supervised by Byron Jones and Stephen Senn) is not a reliable source. Nevertheless, it contains some gems.
  1. About PBE/IBE suggested by the FDA (p.69)
        Other academic resources (Senn, 2000) held that average bioequivalence should suffice based upon grounds of ‘practicality, plausibility, historical adequacy, and purpose’ and ‘because we have better things to do’. Additionally, Senn (2000) notes that statisticians have ‘a bad track record in bioequivalence’, that ‘the literature is full of ludicrous recommendations from statisticians ’, that ‘regulatory recommendations (of dubious validity) have been hastily implemented’, and that ‘practical realities have been ignored’.

  2. About estimating variances (p.190–2)
        It is known (Patterson et al., 1999; Zariffa et al., 2000) that variance estimates generated in bioequivalence studies powered for average bioequivalence are poorly (i.e. imprecisely) characterized, and estimates in excess of the \(\small{\sigma_\textrm{D}>\textrm{cut-off}}\) (Hauck et al., 2000) should be expected due to random chance […]. Increasing sample size does appear to provide some benefit in making these estimates quantitatively more precise […] but the larger sample sizes needed to achieve this are not currently recommended for demonstration of average bioequivalence in moderate variability compounds (FDA Guidance, 2001).
        Thus, as a practical matter, in studies powered for ABE (the current international standard […]), inference on variance estimates, or resulting metrics like IBE and PBE, should be approached with caution.


  3. About covariance structures (p.207)
        Only one REML procedure (UN) was found to yield unbiased estimates in complete data sets and those with missing data. Method-of-moments, as expected, yielded unbiased estimates in complete data sets, but was positively biased in samples with missing data. Bias in method-of-moments (with missing data) and constrained REML procedures increased as drugs become more highly variable and decreased with increasing sample size. Biased method-of-moments estimates in data sets with missing data were greater than those found in CSH REML which were in turn observed to be slightly greater than thos derived using RIS REML. The performance of estimates from FA0(2) REML was questionable. Estimates were positively biased when the true \(\small{\sigma_\textrm{D}^2=0}\) and estimates were negatively biased when \(\small{\sigma_\textrm{D}^2>0}\).

#1 is funny but IMHO, true.
#2 is interesting. The same holds true for RSABE. Even worse for the FDA’s RSABE of NTIDs where a test of swT/swR is part of the procedure. Are the sample sizes large enough?
#3 ‘The performance of estimates from FA0(2) REML was questionable’. Oops! Note that the simulations were performed for 4-period 2-sequence full (‼) replicate designs.

Regrettably I failed to find Senn (2000).*


  • Senn S. Conference Proceedings: Challenging Statistical Issues in Clinical Trials. 2000. Decisions and Bioequivalence.

Dif-tor heh smusma 🖖
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
zizou
★    

Plzeň, Czech Republic,
2020-08-13 23:24
(110 d 21:45 ago)

@ Helmut
Posting: # 21874
Views: 2,411
 

 Partial replicate design: reference(s)?

Dear Helmut,
in this post I mentioned the oldest study with partial replicate design (sequences RRT,RTR,TRR) I know about. The study was conducted at AstraZeneca R&D Lund, Lund, Sweden in 1999.

PDF (opening page linked to Statistical Review).

Best regards,
zizou
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2020-08-14 11:43
(110 d 09:26 ago)

@ zizou
Posting: # 21875
Views: 2,388
 

 Donald’s model

Dear zizou,

» in this post I mentioned the oldest study with partial replicate design (sequences RRT,RTR,TRR) I know about.

THX! Funky model by Donald Schuirmann:

PROC MIXED;
CLASS SUBJ SEQ PER GRP TRT;
MODEL Y =
  GRP SEQ GRP*SEQ PER GRP*PER TRT/DDFM=SATTERTH;
RANDOM SUBJ(GRP*SEQ) SUBJ*TRT(GRP*SEQ);
ESTIMATE 'T VS.R' TRT -1 1/CL ALPHA=0.1;
RUN;

Two things are interesting: No REPEATED statement and the covariance structure is not specified. Hence, SAS applies the default TYPE=VC (variance components). Let’s see what happens in PHX/WNL with the EMA’s data set II (of course, group terms deleted from Donald’s setup).
  1. FDA’s code (2001 BE-stats guidance and 2010 progesterone guidance)
    PE 102.26% (90% CI 97.05–107.76%)
    Warning 11091: Newton's algorithm converged with modified Hessian. Output is suspect.

  2. As above but TYPE=FA0(1) instead of TYPE=FA0(2)
    PE 102.26% (90% CI 97.05–107.76%)
  3. Donald
    PE 102.26% (90% CI 94.37–110.82%)
  4. EMA Method B (implicitly DDFM=CONTAIN)
    PE 102.26% (90% CI 97.32–107.46%)
    Same results with DDFM=SATTERTH cause the data set is complete and balanced.
I think that Donald’s model doesn’t make sense cause the replicative structure of the study is ignored (given, like in the EMA’s Method A and B).

Dif-tor heh smusma 🖖
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
d_labes
★★★

Berlin, Germany,
2020-08-14 14:03
(110 d 07:07 ago)

@ Helmut
Posting: # 21878
Views: 2,356
 

 Donald’s model - model for (logistic) groups

Dear Helmut,

IMHO Donalds model is a variant of the model with logistic groups, employing SAS Proc MIXED instead of Proc GLM.

Regards,

Detlew
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2020-08-14 14:06
(110 d 07:04 ago)

@ d_labes
Posting: # 21879
Views: 2,350
 

 Donald’s model - model for (logistic) groups

Dear Detlew,

THX for the clarification. I don’t speak SASian. ;-)

Dif-tor heh smusma 🖖
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2020-08-14 19:37
(110 d 01:33 ago)

@ Helmut
Posting: # 21881
Views: 2,367
 

 Interlude I (sample sizes, problems & remedies)

Dear all,

while I still hope for a reliable source (i.e., published in a peer-reviewed journal) of the partial replicate, an interlude about sample sizes and more.

Of course, I understand that people prefer three periods over four (less blood volume, lower chance of drop­outs). László Tóthfalusi mentioned (IIRC, »Dissolution Testing, Bio­avail­abi­lity & Bio­equivalence Conference«, Budapest 2006) that power depends on the number of administrations and therefore, if \(\small{N}\) is the sample size of a 2×2×2 crossover, for a 4-period replicate one would need \(\small{n=N/2}\) subjects and for 3-period replicates \(\small{n=3N/4}\). However, this is only approximate because the degrees of freedom are different (see below). Currently, this approximation is implemented – sorry, Dave and Yung-jin – in FARTSSIE and the R-package bear.

   CV design  type    df   n  df   power admins appr match dev part.full
 0.20  2x2x2 conv.   N-2  20  18 0.83468     40                         
 0.20  2x2x4  full 3*n-4  10  26 0.84331     40   10   yes             
 0.20  2x2x3  full 2*n-3  14  25 0.81793     42   16    no   +         
 0.20  2x3x3 part. 2*n-3  15  27 0.84401     45   15   yes             +
 0.25  2x2x2 conv.   N-2  28  26 0.80744     56                         
 0.25  2x2x4  full 3*n-4  14  38 0.81399     56   14   yes             
 0.25  2x2x3  full 2*n-3  22  41 0.83198     66   22   yes             
 0.25  2x3x3 part. 2*n-3  21  39 0.81434     63   21   yes             –
 0.30  2x2x2 conv.   N-2  40  38 0.81585     80                         
 0.30  2x2x4  full 3*n-4  20  56 0.82024     80   20   yes             
 0.30  2x2x3  full 2*n-3  30  57 0.82040     90   30   yes             
 0.30  2x3x3 part. 2*n-3  30  57 0.82040     90   30   yes             =
 0.35  2x2x2 conv.   N-2  52  50 0.80747    104                         
 0.35  2x2x4  full 3*n-4  26  74 0.81090    104   26   yes             
 0.35  2x2x3  full 2*n-3  38  73 0.80082    114   40    no   +         
 0.35  2x3x3 part. 2*n-3  39  75 0.81099    117   39   yes             +
 0.40  2x2x2 conv.   N-2  66  64 0.80525    132                         
 0.40  2x2x4  full 3*n-4  34  98 0.81934    136   34   yes             
 0.40  2x2x3  full 2*n-3  50  97 0.81189    150   50   yes             
 0.40  2x3x3 part. 2*n-3  51  99 0.81940    153   51   yes             +
 0.45  2x2x2 conv.   N-2  82  80 0.80691    164                         
 0.45  2x2x4  full 3*n-4  42 122 0.81823    168   42   yes             
 0.45  2x2x3  full 2*n-3  62 121 0.81220    186   62   yes             
 0.45  2x3x3 part. 2*n-3  63 123 0.81826    189   63   yes             +
 0.50  2x2x2 conv.   N-2  98  96 0.80322    196                         
 0.50  2x2x4  full 3*n-4  50 146 0.81281    200   50   yes             
 0.50  2x2x3  full 2*n-3  74 145 0.80768    222   74   yes             
 0.50  2x3x3 part. 2*n-3  75 147 0.81283    225   75   yes             +
 0.55  2x2x2 conv.   N-2 116 114 0.80386    232                         
 0.55  2x2x4  full 3*n-4  58 170 0.80539    232   58   yes             
 0.55  2x2x3  full 2*n-3  86 169 0.80089    258   88    no   +         
 0.55  2x3x3 part. 2*n-3  87 171 0.80541    261   87   yes             +
 0.60  2x2x2 conv.   N-2 134 132 0.80173    268                         
 0.60  2x2x4  full 3*n-4  68 200 0.80879    272   68   yes             
 0.60  2x2x3  full 2*n-3 100 197 0.80111    300  102    no   +         
 0.60  2x3x3 part. 2*n-3 102 201 0.80880    306  102   yes             +

The approximation works well for the 4-period replicate but not so for the 3-period replicates. Hence, I suggest PowerTOST. In most cases the partial replicate requires more subjects than the 3-period full replicate. :-|

In a nutshell:

 Partial replicate (TRR|RTR|RRT) 

  1. Generally slightly larger sample sizes than full replicates. 〰️
    Since same degrees of freedom, the sample size is a multiple of three instead of two.
  2. Estimation of swR / CVwR.
  3. Estimation of swT / CVwT not possible.
  4. The EMA’s “all fixed effects” model for ABE.
  5. Sometimes convergence issues with the FDA’s covariance structure of the mixed effects model for ABE.* Changing from FA0(2) to FA0(1) or CSH generally – not always! – helps. Should be specified in the SAP. Rarely convergence is not achieved with any specification. 🆘
  6. The EMA’s ABEL (CVwR >30%).
  7. The FDA’s RSABE (swR ≥0.294).

 3-period full replicates (TRT|RTR or TRR|RTT) 

  1. Generally slightly smaller sample sizes than partial replicate. 〰️
    Since same degrees of freedom, the sample size is a multiple of two instead of three.
  2. Estimation of swR / CVwR.
  3. Estimation of swT / CVwT.
  4. The EMA’s “all fixed effects” model for ABE.
  5. The FDA’s covariance structure of the mixed effects model for ABE.
  6. The EMA’s ABEL (CVwR >30%).
  7. The FDA’s RSABE (swR ≥0.294).

#1: Not a big deal; if sample sizes are different, the difference is pretty small.
#3: In a pilot study CVwT is not only nice to know but useful. If CVwT < CVwR you get an incentive in planning the pivotal study (scaling depends on CVwR but the BE assessment on the pooled CVw). If CVwT > CVwR you can account for that and increase the sample size accordingly. If the pilot study was performed in the partial replicate design you have to assume that CVwT = CVwR. If CVwT < CVwR you waste money. If CVwT > CVwR your study will be underpowered. For a pivotal study you can approach the WHO for reference-scaling of AUC (cause a 4-period full replicate is recommended in the guidance).
#5: Can be a show stopper in the partial replicate design. There is a slight risk (ABE for the FDA using FA0(2) acc. to the guidance and no convergence) that the PK metric in question cannot be assessed for ABE at all.

 Given all that, I don’t see any justification
for using the partial replicate design. 

If you know one, please let me know.


  • The model tries to estimate \(\small{s_\textrm{wT}^2}\) though the test was only administered once. Hence, the model is over-spe­ci­fied for this design. Even if convergence is achieved, the estimate is nonsense and differs (contrary to \(\small{s_\textrm{wR}^2}\)) between software packages. Only the total variance \(\small{s_\textrm{T}^2=s_\textrm{wT}^2+s_\textrm{bT}^2}\) could be estimated but not done in the model.

R-script for the table:

library(PowerTOST)
balance <- function(x, y) {
  return(y * (x %/% y + as.logical(x %% y)))
}
designs <- known.designs()[c(3, 8, 7, 10), 2:3]
type    <- c("conv.", "full", "full", "part.")
eval(parse(text = designs$df[1]))
CV  <- seq(0.2, 0.6, 0.05)
res <- data.frame(CV = rep(CV, each = 4), design = designs[, 1], type = type,
                  df.1 = designs[, 2], n = NA, df.2 = NA, power = NA,
                  admins = NA, appr = "", match = "", dev = "", part.full = "")
for (j in 1:nrow(res)) {
  tmp <- sampleN.TOST(CV = res$CV[j], design = res$design[j],
                      print = FALSE, details = FALSE)
  res$n[j]     <- tmp[["Sample size"]]
  res$power[j] <- signif(tmp[["Achieved power"]], 5)
  n            <- res$n[j]
  nseq         <- as.numeric(substr(res$design[j], 3, 3))
  nper         <- as.numeric(substr(res$design[j], 5, 5))
  if (res$design[j] == "2x2x2") N <- n
  if (!res$design[j] == "2x2x2") {
    if (nper == 4) res$appr[j] <- balance(N/2, nseq)
    if (nper == 3) res$appr[j] <- balance(3*N/4, nseq)
    if (res$appr[j] == res$n[j]) {
      res$match[j] <- "yes"
    } else {
      res$match[j] <- "no"
      ifelse (res$appr[j] < res$n[j], res$dev[j] <- "\u2013", res$dev[j] <- "+")
    }
    if (res$design[j] == "2x2x3") n.3full <- n
    if (res$design[j] == "2x3x3") {
      if (n < n.3full)  res$part.full[j] <- "\u2013"
      if (n == n.3full) res$part.full[j] <- "="
      if (n > n.3full)  res$part.full[j] <- "+"
    }
  }
  res$admins[j] <- nper*n
  df            <- designs$df[which(designs$design == res$design[j])]
  res$df.2[j]   <- eval(parse(text = df))
}
names(res)[c(4, 6)] <- rep("df", 2)
res[which(res[, 4] == "n-2"), 4] <- "N-2"
print(res, row.names = FALSE)


R-script to simulate 1,000 studies; CV 20–60%, T/R-ratio 0.85–0.95, power 80–90%:

library(PowerTOST)
set.seed(1234567)
sims   <- 1000
CV     <- runif(sims, 0.20, 0.60)
theta0 <- runif(sims, 0.85, 0.95)
target <- runif(sims, 0.80, 0.90)
res    <- data.frame(f = rep(NA, sims), p = NA)
for (j in 1:sims) {
  res$f[j] <- sampleN.TOST(CV = CV[j], theta0 = theta0[j], design = "2x2x3",
                           targetpower = target[j], print = FALSE)[["Sample size"]]
  res$p[j] <- sampleN.TOST(CV = CV[j], theta0 = theta0[j], design = "2x3x3",
                           targetpower = target[j], print = FALSE)[["Sample size"]]
}
bp <- boxplot(res, plot = FALSE)
plot(c(0.5, 2.5), range(bp$stats[1, ], bp$stats[5, ]), type = "n", axes = FALSE,
     xlab = "3-period replicate designs", ylab = "sample size",
     main = paste(sims, "studies simulated for ABE"), cex.main = 1.2, font.main = 1)
abline(h = axTicks(2, log = FALSE), col = "lightgrey", lty = 3)
axis(1, at = 1:2, labels = c("full", "partial"), tick = FALSE)
axis(2, las = 1)
bxp(bp, las = 1, outline = FALSE, boxwex = 0.4, add = TRUE, ann = FALSE,
    boxfill = "bisque", medcol = "blue", axes = FALSE, frame.plot = TRUE)
text(rep(1.25, 5), bp$stats[, 1], labels = bp$stats[, 1], pos = 4)
text(rep(2.25, 5), bp$stats[, 2], labels = bp$stats[, 2], pos = 4)


Dif-tor heh smusma 🖖
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2020-08-19 21:31
(104 d 23:38 ago)

@ Helmut
Posting: # 21889
Views: 1,969
 

 Interlude II (simulations)

Dear all,

I simulated 500 data sets in the partial replicate design with \(\small{s_\textrm{wT}^2=s_\textrm{wR}^2=0.086\: (CV_\textrm{w}\approx 29.97\%),}\) \(\small{s_\textrm{bT}^2=s_\textrm{bR}^2=0.172\: (CV_\textrm{b}\approx 43.32\%),}\) \(\small{\rho=1},\) \(\small{\theta_0=1},\) i.e., no subject-by-formulation interaction. With \(\small{n=24}\) subjects 82.3797% power to demonstrate ABE. Evaluation in Phoenix/WinNonlin 8.1 with the FDA’s covariance structure FA0(2). Singularity tolerance and convergence criterion 1E-12 (instead of 1E-10), maximum iterations 250 (instead of 50).
If you want to try it in SAS or any other software: The data sets in CSV-format.

In 403 (80.6%) of the data sets PHX issued at least one warning.
In 56 (11.2%) of the data sets PHX threw this:

Negative final variance component. Consider omitting this VC structure.

Well roared lion. The model reached for the stars (namely \(\small{s_\textrm{wT}^2}\)). We know that we can get only the total variability (i.e., \(\small{s_\textrm{T}^2=s_\textrm{wT}^2+s_\textrm{bT}^2}\)) like in a parallel design where the within-subject variance (as well as the between variance) is not accessible as well. Amazingly in the other cases PHX got an estimate though it’s nonsense, of course.

In 340 (68%) of the data sets I was told:

Model may be over-specified. A simpler model could be tried.

Oh yes! »May be« is an euphemism. Actually it is. Not only in some of the data sets but in all.

In 333 (66.6%) of the data sets PHX threw this:

Newton's algorithm converged with modified Hessian. Output is suspect.

How I love to be told that results are suspect. Will assessors love that as well? Well, in PHX it’s hidden in the ‘Core Output’ and the ‘Warnings and Errors’. In SAS in a log-file… Will it be shown in your fancy output? Not necessarily.

79.6% of the data sets passed BE. That’s only slightly lower than expected and likely due to the small number of simulations (10,000 are running).
How close are the estimates to the targets?

                s²wR     s²bR     s²T       S×F      PE
─────────────────────────────────────────────────────────
target         0.08600  0.17200  0.25800  0        100.00
mean estimate  0.08733  0.16923  0.22655  0.01090   98.48
%RE            +1.54%   –1.61%   –12.19%   –       –1.52%

Fine with me, except \(\small{s_\textrm{T}^2}\) which is not directly estimated (see this lengthy thread for a promising alternative) but as the sum of two doubtful estimates (\(\small{s_\textrm{wT}^2,s_\textrm{bT}^2}\)).

We see also that the optimizer is fine in estimating CVwR but desperate with CVwT (only 437 values). The target was 29.9677% for both.

       min    QI     med    QIII   max
───────────────────────────────────────
CVwR  14.25  26.75  30.11  33.20  46.13
CVwT   1.33  18.29  23.23  28.05  45.33


I evaluated the data sets with other covariance structures as well. Seems that FA0(1) is the winner.

Convergence          FA0(2)      FA0(1)        CS
─────────────────────────────────────────────────────
Achieved          160 (32.0%)  500 (100%)  500 (100%)
Modified Hessian  340 (68.0%)     –           –

Warnings                       FA0(2)      FA0(1)      CS
─────────────────────────────────────────────────────────────
Modified Hessian            333 (66.6%)     –          –
Negative variance component  56 (11.2%)  25 (5.0%)  56 (11.2%)
Both                         14 ( 2.8%)     –          –

As long as we achieve convergence, it doesn’t matter (we have seen nasty data sets in the past, where FA0(2) didn’t converge). Perhaps as long as the data set is balanced and/or does not contain ‘outliers’, all is good. At the end of the day we are interested in the 90% CI. I compared the results obtained with FA0(1) and CS to the guidances’ FA0(2). Up to the 4th decimal (rounded to percent, i.e., 6–7 significant digits) the CI was identical in all cases. Only when I looked at the 5th decimal for both covariance structures, 1/500 differed (the CI was wider). Since all guidelines require rounding to the 2nd decimal, that’s not relevant.

I’m not a friend of the EMA’s ‘all effects fixed’ model because it assumes identical variances of T and R (which has be shown to be wrong in many full replicate studies). But, of course, no issues with convergence in this simple linear model.

My original simulation code contained a stupid error (THX to Detlew for detecting it!) which lead to an extreme S×F-interaction. Example of one data set where the optimizer was in deep trouble. The default maximum iterations in PHX/WNL are 50. I got:

max.iter    s²wR     %RE  -2REML LL   AIC    BIC       df         90% CI
────────────────────────────────────────────────────────────────────────────
    50    0.084393  –1.87  39.368   61.368  85.455  22.10798  82.212–111.084
   250    0.085094  –1.05  39.345   61.345  85.431  22.18344  82.223–111.070
 1,250    0.085271  –0.85  39.339   61.339  85.425  22.20523  82.225–111.066
 6,250    0.085309  –0.80  39.338   61.338  85.424  22.20991  82.226–111.066
31,250    0.085317  –0.79  39.338   61.338  85.424  22.21032  82.226–111.066

A lament in all cases:

Failed to converge in allocated number of iterations. Output is suspect.

Note that the degrees of freedom increase with the number of iterations and hence, the CI narrows. Now I understand why Health Canada requires that the optimizer’s constraints are stated in the SAP.

[image]


Welcome to the hell of mixed effects modeling. 👿


Edit 1: Results of a large data set (10,000 simulations, 20.5 MB in CSV-format). 80.95% passed BE in all setups. Relevant estimates were identical and pretty close to the targets:

                s²wR      PE
──────────────────────────────
target         0.08600  100.00
mean estimate  0.08584   99.97
%RE            –0.19%   –0.03%

Convergence        FA0(2)  FA0(1)   CS
───────────────────────────────────────
Achieved          30.14%   99.97%  100%
Modified Hessian  69.83%     –      –
> max. iter.       0.03%    0.03%   –

Warnings                    FA0(2)  FA0(1)   CS
─────────────────────────────────────────────────
Modified Hessian            68.75%    –      –
Negative variance component  9.01%  3.82%  11.15%
Both                         2.22%  0.06%    –

Given all that, I would opt for FA0(1).


Edit 2: I manipulated the small data sets. Removed subject 24 to make the study imbalanced, removed the last period (T) of subject 23 to make it incomplete. Multiplied T of subject 1 with 5–10 to mimic an ‘outlier’. Only 15.4% of studies passed. Yep, even a single outlier might be the killer. 381 warnings by FA0(2), 2 by FA0(1), and 7 by CS.

Dif-tor heh smusma 🖖
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
PharmCat
★    

Russia,
2020-08-19 22:26
(104 d 22:44 ago)

@ Helmut
Posting: # 21890
Views: 1,953
 

 Interlude II (simulations)

Hi Helmut!

using DataFrames, CSV, ReplicateBE, LinearAlgebra

path      = dirname(@__FILE__)
dfa       = CSV.File(path*"/sim500-2x3x3.csv") |> DataFrame
dfa.logpk = log.(dfa.PK)

res = Vector{Any}(undef, 500)
pd  = Vector{Any}(undef, 500)
os  = Vector{Any}(undef, 500)
for i in 1:500
    df = filter(r -> i == r.set , dfa)
    res[i] = ReplicateBE.rbe!(df, dvar = :logpk, subject = :subject, formulation = :treatment, period = :period, sequence = :sequence, g_tol = 1e-10, singlim = 1e-12)
    pd[i]  = isposdef(Symmetric(res[i].result.H))
    os[i]  = ReplicateBE.optstat(res[i])
    println(i)
end
println("Positive definite: ", sum(pd), "(",sum(pd)/500*100,"%)")
println("Converged: ", sum(os), "(",sum(os)/500*100,"%)")


Positive definite: 332(66.4%)
Converged: 500(100.0%)

I can make table with results if necessary.
Activity
 Admin contact
21,232 posts in 4,427 threads, 1,481 registered users;
online 5 (1 registered, 4 guests [including 2 identified bots]).
Forum time: Wednesday 20:10 CET (Europe/Vienna)

A big data-analyst is an expert
in producing misleading conclusions from huge datasets.
It is much more efficient to use a statistician,
who can do the same with small ones.    Stephen Senn

The Bioequivalence and Bioavailability Forum is hosted by
BEBAC Ing. Helmut Schütz
HTML5