Helmut
★★★
avatar
Homepage
Vienna, Austria,
2019-10-16 13:14

Posting: # 20690
Views: 957
 

 Missing periods in replicate designs [RSABE / ABEL]

Dear all,

sometimes I see reports with a strange pattern of missing periods, e.g., subjects without the 1st period, subjects with only the 1st and the 4th, etc. Since I have only the statistical part of the report, I have no clue what was going on.
For the FDA’s RSABE only subjects with complete data (all periods) are used. But this is not the case for the EMA’s ABEL (all subjects with at least one T and R treatment for the calculation of the CI, all subjects with two R treatments for the calculation of CVwR).

I would say:
  • If a subject does not show up or withdraws consent before the first dose, that’s the end of the story.
  • If a subject drops out in later periods, end of the story as well. Why did they come back?
Did you see cases like this? And if yes, do you know why?
I saw even cases where a subject in sequence TRTR had data only of periods 1 & 3. Useless.
Another case: 44 subjects had two administrations of T and R. However, 15 had two administrations of T and 39 two administrations of R. Chance – or what?

Cheers,
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. ☼
Science Quotes
ElMaestro
★★★

Belgium?,
2019-10-16 15:10

@ Helmut
Posting: # 20693
Views: 898
 

 Missing periods in replicate designs

Hi Hötzi,

» I would say:
  • If a subject does not show up or withdraws consent before the first dose, that’s the end of the story.
    »
  • If a subject drops out in later periods, end of the story as well. Why did they come back?
Did you see cases like this? And if yes, do you know why?

Not a definitive answer, but at some CROs they have SOPs in place to the effect of allowing subjects to come back after they have skipped a period , or fractions of one. This, I think, relates originally to FDA's data driven policies. A subject can walkout on her/his own initiative without even stating a reason, that's how GCP works. There is no clause saying she/he can't come back.
I know, this is messy, but that's the way it is in some clinics.

The alternative may be worse, depending on how you look at it: If a subject misses an ambulatory pk-sample ("I forgot", "I sat stuck in a traffic jam", "My parrot suffered an anxiety attack", "Don't you fucking ask me what I did yesterday, it's none of your business", "I had to watch Conchita Wurst win the Grand Prix" etc.) should she/he then be considered completely out?

I could be wrong, but...
Best regards,
ElMaestro
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2019-10-17 10:22

@ ElMaestro
Posting: # 20700
Views: 860
 

 Missing samples

Hi ElMaestro,

» This, I think, relates originally to FDA's data driven policies. A subject can walkout on her/his own initiative without even stating a reason, that's how GCP works. There is no clause saying she/he can't come back.
» I know, this is messy, but that's the way it is in some clinics.

I see. Since the data are useless for the FDA’s reference-scaling maybe this was a trick-shot of the CRO to milk the sponsor.

» The alternative may be worse, depending on how you look at it: If a subject misses an ambulatory pk-sample […] should she/he then be considered completely out?

I like your examples! That’s an area of improvement which requires a good amount of intellectual horsepower. Not easy but the idea is to come up with rules specifying how many missings (and where) in the profile will likely lead to unreliable estimates of a given PK metric.

Cheers,
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. ☼
Science Quotes
d_labes
★★★

Berlin, Germany,
2019-10-17 17:22

@ Helmut
Posting: # 20701
Views: 828
 

 Missing periods in replicate designs

Dear Helmut!

» sometimes I see reports with a strange pattern of missing periods, e.g., subjects without the 1st period, subjects with only the 1st and the 4th, etc. ...
» Did you see cases like this? And if yes, do you know why?

Yes, I saw such cases in times I was active.
F.i. a subject refuses consent after the 2nd period but came back due to invention of the investigator (by phone) for the 4th period (arguments: "you will not get payed if you refuse, only if you come back" or "Please, please be so kind to continue because otherwise we have not the scientific success we expected" and so on. See ElMaestros post for other examples.)

What to do with such data? DUNO exactly.
You yourself have described what to do for EMA ABEL or FDA RSABE.
Any case left?

Regards,

Detlew
ElMaestro
★★★

Belgium?,
2019-10-17 20:17

@ d_labes
Posting: # 20702
Views: 813
 

 Missing periods in replicate designs

Hi d_labes,

» "you will not get payed if you refuse, only if you come back"
» "Please, please be so kind to continue because otherwise we have not the scientific success we expected"

§4.8.3: Neither the investigator, nor the trial staff, should coerce or unduly influence a subject to participate or to continue to participate in a trial.


:-D:-D:-D:-D:-D

I could be wrong, but...
Best regards,
ElMaestro
d_labes
★★★

Berlin, Germany,
2019-10-18 11:48

@ ElMaestro
Posting: # 20703
Views: 779
 

 Missing periods in replicate designs

Dear ElMaestro,

» ...
» §4.8.3: Neither the investigator, nor the trial staff, should coerce or unduly influence a subject to participate or to continue to participate in a trial.

Was of course not an unduly influence. Only a friendly talk by phone to obtain some informations about the reasons and circumstances of refusing the continuation.

Regards,

Detlew
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2019-10-19 14:30

@ d_labes
Posting: # 20706
Views: 726
 

 Missing periods in replicate designs

Dear Detlew,

» You yourself have described what to do for EMA ABEL or FDA RSABE.

Yep.

» Any case left?

Nope. I was just wondering what could be the cause of such results.

Cheers,
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. ☼
Science Quotes
Astea
★    

Russia,
2019-10-20 14:46

@ Helmut
Posting: # 20709
Views: 701
 

 Missing periods in replicate designs: save the data?

Dear smart people!

I've got two questions on this topic, that could be logically combined into the one philosophical: should we try to keep as much data as possible for the analysis?

1). How would you advice to deal with subjects, who have only one (2,3..) points over LLOQ in one of the periods? According to EMA it is possible to exclude subjects with AUC of reference product less than 5% of geometric mean AUC. Should we exclude all the data of such a subject or can we remain data from other periods?
Example: acetylcalycic acid (ASA) in enteric-coated form has a wide-range Tmax in about 2 to 7 hours with extremely rapid conversion to salycic acid. So PK profiles look like "zero-zero-vertical line-zero-zero"... For catching it one has to plan a great number of sample points and use appropriate stabilization procedure. Even in these case there could be the uppermentioned problems.


2). What was the real reason for FDA to develop an algorithm for NTIDs with only complete data? As Helmut mentioned in the post, theoretically it is possible to use all the data even with incomplete data. Why then FDA just throw data of subjects with incomplete data to the bin? Is not it unethical? (I can't understand this point)

"We are such stuff as dreams are made on, and our little life, is rounded with a sleep"
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2019-10-20 15:23

@ Astea
Posting: # 20710
Views: 699
 

 Missing periods in replicate designs: save the data?

Hi Nastia,

» Dear smart people!

     ^^^^ Are you talking to me?


» should we try to keep as much data as possible for the analysis?

In principle yes – as long as the outcome is meaningful. ;-)

» 1). How would you advice to deal with subjects, who have only one (2,3..) points over LLOQ in one of the periods? […]

Tricky – IMHO, case by case (should be laid down in an SOP or the SAP, of course). IIRC, Health Canada had a rule that 1 (one!) concentration is sufficient for Cmax and 3 (oh dear!) fo AUC. Gone with the wind. THX, HC.

» 2). What was the real reason for FDA to develop an algorithm for NTIDs with only complete data?

No idea. The same is applicable to all RSABE-methods of the FDA.

» […] theoretically it is possible to use all the data even with incomplete data.

Sure.

» Why then FDA just throw data of subjects with incomplete data to the bin?

Again – no idea.

» Is not it unethical? (I can't understand this point)

IMHO, it is and we are not alone with this conclusion.*

If SABE is applied, subjects with one missing R observation should be eliminated […]. This is unprecedented in our experience in a regulated bioequivalence setting. Traditionally, one does not exclude data unless there is a scientifically or clinically valid reason to do so. However, with the current draft guidance from FDA for progesterone bioequivalence, this appears to be the immediate approach to be applied for SABE.



  • Patterson SD, Jones B. Viewpoint: observations on scaled average bioequivalence. Pharmaceut Stat. 2012; 11(1):1–7. doi:10.1002/pst.498.

Cheers,
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. ☼
Science Quotes
Astea
★    

Russia,
2019-10-21 01:59

@ Helmut
Posting: # 20711
Views: 680
 

 Missing periods in replicate designs: kill the data?

Dear Helmut and other smart people!

» Are you talking to me?
Hey, you! Yes, you!

» Tricky – IMHO, case by case (should be laid down in an SOP or the SAP, of course). IIRC, Health Canada had a rule that 1 (one!) concentration is sufficient for Cmax and 3 (oh dear!) fo AUC. Gone with the wind. THX, HC.

Suppose we have only one measurable concentration and it is as high as Cmax of other periods. Even in this case formally we can calculate AUC for linear trapezoidal rule (depending on the rule for BLOQ data of course). And it can be more than 5 percent of geometric mean of other AUC in this period. Leave it or waste it - any choice will be wrong :no:
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2019-10-21 11:37

@ Astea
Posting: # 20712
Views: 639
 

 Meaningful data?

Dear Nastia,

» Suppose we have only one measurable concentration and it is as high as Cmax of other periods. Even in this case formally we can calculate AUC for linear trapezoidal rule […]. And it can be more than 5 percent of geometric mean of other AUC in this period. Leave it or waste it - any choice will be wrong :no:

Well, that’s why I wrote above

» » […] as long as the outcome is meaningful. ;-)

We have to go out on a limb to call a triangle a “curve”.

Cheers,
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. ☼
Science Quotes
Mahmoud
☆    

Jordan,
2019-11-06 11:41

@ Helmut
Posting: # 20753
Views: 253
 

 Missing periods in replicate designs

Dear all
=====
For missing data in Be studies in certain period

if you need to take into account the missing data use proc mixed in SAS with dffm=Kr

otherwise use proc glm in SAS, this procedure do not take into account the missing data
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2019-11-06 12:49

@ Mahmoud
Posting: # 20754
Views: 250
 

 Kenward-Roger?

Hi Mahmoud,

» if you need to take into account the missing data use proc mixed in SAS with dffm=Kr

From a theoretical perspective, maybe. The Kenward-Roger approximation recovers more information from the data, higher degrees of freedom and hence, results in a narrower confidence interval than with Satterthwaite’s degrees of freedom. Since the latter is explicitly recommended in the progesterone guidance I have some doubts whether the FDA would accept that.

» otherwise use proc glm in SAS, this procedure do not take into account the missing data

Sure.

Cheers,
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. ☼
Science Quotes
PharmCat
☆    

Russia,
2019-11-06 14:43

@ Helmut
Posting: # 20755
Views: 242
 

 Kenward-Roger?

» From a theoretical perspective, maybe. The Kenward-Roger approximation recovers more information from the data, higher degrees of freedom and hence, results in a narrower confidence interval than with Satterthwaite’s degrees of freedom. Since the latter is explicitely recommended in the progesterone guidance I have some doubts whether the FDA would accept that.

I thought that Kenward-Roger provide the same DF as Satterthwaite's for one-dimension effects, so as CI for coefficient is one-dimension hypothesis DF should be the same, as it describes in reference paper, may be SAS make any corrections, I don't know... Is any comparations anywhere?
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2019-11-08 20:57

@ PharmCat
Posting: # 20771
Views: 188
 

 Kenward-Roger ≥ Satterthwaite

Hi PharmCat,

» I thought that Kenward-Roger provide the same DF as Satterthwaite's for one-dimension effects, so as CI for coefficient is one-dimension hypothesis DF should be the same, as it describes in reference paper, may be SAS make any corrections, I don't know...

Try this one:

library(replicateBE)
ds  <- substr(grep("rds", unname(unlist(data(package = "replicateBE"))),
                   value = TRUE), start = 1, stop = 5)
res <- data.frame(rds = 1:length(ds), df.Satt = NA, df.KR = NA)
for (j in seq_along(ds)) {
  res$df.Satt[j] <- method.B(option = 1, print = FALSE, details = TRUE,
                             data = eval(parse(text = ds[j])))$DF
  res$df.KR[j]   <- method.B(option = 3, print = FALSE, details = TRUE,
                             data = eval(parse(text = ds[j])))$DF
}
res[, 2:3] <- signif(res[, 2:3], 5)
print(res, row.names = FALSE)


The EMA’s Method B evaluated by lmer() of package lmerTest. Kenward-Roger’s degrees of freedom ≥ Satterthwaite’s.

Cheers,
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. ☼
Science Quotes
PharmCat
☆    

Russia,
2019-11-08 22:08

@ Helmut
Posting: # 20773
Views: 179
 

 Kenward-Roger ≥ Satterthwaite

» The EMA’s Method B evaluated by lmer() of package lmerTest. Kenward-Roger’s degrees of freedom ≥ Satterthwaite’s.

May be it's specific of realization in lmer. I can't understand what's under the hood :-D I will make test in SAS.
mittyri
★★  

Russia,
2019-11-08 22:59

@ PharmCat
Posting: # 20774
Views: 174
 

 Kenward-Roger - links

Hi PharmCat,

» I thought that Kenward-Roger provide the same DF as Satterthwaite's for one-dimension effects, so as CI for coefficient is one-dimension hypothesis DF should be the same, as it describes in reference paper, may be SAS make any corrections, I don't know...

"KR modify the statistic F to improve the small sample properties by approximating the distribution of F by an Fd,m distribution, and they also provide a method for calculating the denominator degrees of freedom m. The fundamental idea is to calculate the approximate mean and variance of their statistic and then match moments with an F distribution to obtain the denominator degrees of freedom."
from here

» Is any comparations anywhere?

yes, plenty of. For example here
Also citing Kuznetsova et al.: "From our practice, we observed that the p values that the approximation methods provide are generally very close to each other. Schaalje, McBride, and Fellingham (2002) performed a number of simulations in order to investigate the appropriateness of the approximation methods. They discovered that complexity of the covariance structures, sample size and imbalance affect the performance of both approximations. However, these factors affect the Satterthwaite’s method more than the Kenward-Roger’s."

Kind regards,
Mittyri
PharmCat
☆    

Russia,
2019-11-09 00:51

@ mittyri
Posting: # 20775
Views: 163
 

 Kenward-Roger - links

I look this or here.

"Lemma 7.2.1 When l = 1, the Satterthwaite, the K-R and the proposed methods give
the same estimate of the denominator degrees of freedom"
p.98 proof in source.

For coefficient estimate l = 1, and thou I thought they should be equal (and the really equal in SAS).

But when DDFM=KENWARDROGER is set - variance-covariance matrix of the fixed effects is corrected too and resulting CI would be differ - this I didn’t take it into account.
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2019-11-09 16:22

@ mittyri
Posting: # 20778
Views: 101
 

 degrees of freedom

Hi mittyri and all other nerds,

a rant from Douglas Bates (maintainer of lme4) reproduced in all of its beauty:

Users are often surprised and alarmed that the summary of a linear mixed model fit by lmer provides estimates of the fixed-effects para­meters, standard errors for these parameters and a t-ratio but no p-values. Similarly the output from anova applied to a single lmer model provides the sequential sums of squares for the terms in the fixed-effects specification and the corresponding numerator degrees of freedom but no denominator degrees of freedom and, again, no p-values.

Because they feel that the denominator degrees of freedom and the corresponding p-values can easily be calculated they conclude that failure to do this is a sign of inattention or, worse, incompetence on the part of the person who wrote lmer (i.e. me).

Perhaps I can try again to explain why I don’t quote p-values or, more to the point, why I do not take the “obviously correct” approach of attempting to reproduce the results provided by SAS. Let me just say that, although there are those who feel that the purpose of the R Project – indeed the purpose of any statistical computing whatsoever – is to reproduce the p-values provided by SAS, I am not a member of that group. If those people feel that I am a heretic for even suggesting that a p-value provided by SAS could be other than absolute truth and that I should be made to suffer a slow, painful death by being burned at the stake for my heresy, then I suppose that we will be able to look forward to an exciting finale to the conference dinner at UseR!2006 next month. (Well, I won’t be looking forward to such a finale but the rest of you can.)

As most of you know the t-statistic for a coefficient in the fixed-effects model matrix is the square root of an F statistic with 1 numerator degree of freedom so we can, without loss of generality, concentrate on the F statistics that were present in the anova output. Those who long ago took courses in “analysis of variance” or “experimental design” that concentrated on designs for agricultural experiments would have learned methods for estimating variance components based on observed and expected mean squares and methods of testing based on “error strata”. (If you weren’t forced to learn this, consider yourself lucky.) It is therefore natural to expect that the F statistics created from an lmer model (and also those created by SAS PROC MIXED) are based on error strata but that is not the case.

The parameter estimates calculated by lmer are the maximum likelihood or the REML (residual maximum likelihood) estimates and they are not based on observed and expected mean squares or on error strata. And that’s a good thing because lmer can handle unbalanced designs with multiple nested or fully crossed or partially crossed grouping factors for the random effects. This is important for analyzing data from large observational studies such as occur in psychometrics.

There are many aspects of the formulation of the model and the calculation of the parameter estimates that are very interesting to me and have occupied my attention for several years but let’s assume that the model has been specified, the data given and the parameter estimates obtained. How are the F statistics calculated? The sums of squares and degrees of freedom for the numerators are calculated as in a linear model. There is a slot in an lmer model that is similar to the “effects” component in a lm model and that, along with the “assign” attribute for the model matrix provides the numerator of the F ratio. The denominator is the penalized residual sum of squares divided by the REML degrees of freedom, which is n-p where n is the number of observations and p is the column rank of the model matrix for the fixed effects.

Now read that last sentence again and pay particular attention to the word “the” in the phrase “the penalized residual sum of squares”. All the F ratios use the same denominator. Let me repeat that – all the F ratios use the same denominator. This is why I have a problem with the assumption (sometimes stated as more that just an assumption – something on the order of “absolute truth” again) that the reference distribution for these F statistics should be an F distribution with a known numerator degrees of freedom but a variable denominator degrees of freedom and we can answer the question of how to calculate a p-value by coming up with a formula to assign different denominator degrees of freedom for each test. The denominator doesn’t change. Why should the degrees of freedom for the denominator change?

Most of the research on tests for the fixed-effects specification in a mixed model begin with the assumption that these statistics will have an F distribution with a known numerator degrees of freedom and the only purpose of the research is to decide how to obtain an approximate denominator degrees of freedom. I don’t agree.

There is one approach that I think may be fruitful and that I am currently pursuing. The penalized least squares formulation of a mixed-effects model shows that the residual sum of squares in actually a penalized residual sum of squares and there is a quantity that behaves like the degrees of freedom for a penalized least squares problem. I will insert code to calculate this and see how that behaves in simulation.

For the time being, I would recommend using a Markov Chain Monte Carlo sample (function mcmcsamp) to evaluate the properties of individual coefficients (use HPDinterval or just summary from the coda package). Evaluating entire terms is more difficult but you can always calculate the F ratio and put a lower bound on the denominator degrees of freedom.

If anyone wants to contribute code to calculate the “obviously correct” denominator degrees of freedom from SAS I will incorporate it. However, be warned that the penalized least squares approach and sparse matrix methods used in lmer will require considerable translation from the formulas which typically occur in papers on this topic. Generally those formulas involve the inverse of an n by n matrix where n is the number of observations and you really, really don’t want to try to do that when you have a couple of million observations.


Cheers,
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. ☼
Science Quotes
PharmCat
☆    

Russia,
2019-11-09 19:11

@ Helmut
Posting: # 20779
Views: 90
 

 degrees of freedom

» Hi mittyri and all other nerds,

Hi all again :-D from this part holy war began usually :-D

I really love how Douglas writes and this is one more of his explanation.

And I fully support this point of view and concept of statistical purity (lme4, MixedModels.jl ets): you have modeling results and then do what you want.

Problem is that we have real "degrees of freedom police" :-D

I think that more compromised and more conservative is a "contain" DF = N - rank(XZ), but FDA wrote Satterthwaite.
Activity
 Thread view
Bioequivalence and Bioavailability Forum |  Admin contact
19,975 posts in 4,227 threads, 1,373 registered users;
online 8 (0 registered, 8 guests [including 5 identified bots]).
Forum time (Europe/Vienna): 01:20 CET

The fundamental cause of trouble in the world today is
that the stupid are cocksure
while the intelligent are full of doubt.    Bertrand Russell

The BIOEQUIVALENCE / BIOAVAILABILITY FORUM is hosted by
BEBAC Ing. Helmut Schütz
HTML5