Helmut
★★★
avatar
Homepage
Vienna, Austria,
2024-04-19 03:26
(235 d 01:55 ago)

Posting: # 23958
Views: 3,377
 

 Group-by-Treatment Interaction: the final word? [General Sta­tis­tics]

Dear all,

maybe this article is of interest:

Helmut Schütz , Divan A Burger , Erik Cobo , David D Dubins , Tibor Farkás, Detlew Labes , Benjamin Lang, Jordi Ocaña, Arne Ring , Anastasia Shitova , Volodymyr Stus, Michael Tomashevskiy. Group-by-Treatment Interaction Effects in Comparative Bioavailability Studies. AAPS J. 2024; 26(3): 50. doi:10.1208/s12248-024-00921-x. [image] Open Access. [image] Supplementary Material.


⅔ of the authors are members of the forum... :-D

I’m not sure whether the results of the article will change the respective section of ICH M13A, which in the draft states:

Sample size requirements and/or study logistics may necessitate studies to be conducted with groups of subjects. The BE study should be designed to minimise the group effect in the study. The combination of multiple factors may complicate the designation of group.
BE should be determined based on the overall treatment effect in the whole study population. In general, the assessment of BE in the whole study population should be done without including the Group by Treatment interaction term in the model, but applicants may also use other pre-specified models, as appropriate. However, the appropriateness of the statistical model should be evaluated to account for the multi-group nature of the BE study. Applicants should evaluate potential for heterogeneity of treatment effect across groups, i.e., Group by Treatment interaction. If the Group by Treatment interaction is significant, this should be reported and the root cause of the Group by Treatment interaction should be investigated to the extent possible. Substantial differences in the treatment effect for PK parameters across groups should be evaluated. Further analysis and interpretation may be warranted in case heterogeneity across groups is observed.
(my emphasis)

If not, at least you can use the article in an argument with regulators… See also the meta-study in this article, which I regularly update with new data.

Dif-tor heh smusma 🖖🏼 Довге життя Україна! [image]
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2024-09-20 15:30
(80 d 13:51 ago)

@ Helmut
Posting: # 24203
Views: 1,810
 

 Group-by-Treatment Interaction: FDA

Dear all,

the FDA was definitely not happy with our article. Why am I not surprised?

Sun W , Alosh M, Schuirmann DJ, Grosser S. Letter to the Editor on “Group‑by‑Treatment Interaction Effects in Comparative Bio­avail­ability Studies”. AAPS J. 2024; 26(5): 101. doi:10.1208/s12248-024-00972-0.


Disclaimer The views expressed in this article represent the opinions of the authors, and do not represent the views and/or policies of the U.S. Food and Drug Administration.

At least.

Common procedure, e.g., in ‘Science’, ‘Statistics in Medicine’, ‘Journal of the American Statistical Association’, ‘Biometrics’, ‘British Medical Journals’ (a rant):
  1. An article is published.
  2. Some don’t agree, draft a ‘Letter to the Editor’, and submit it for review.
  3. If the editor deems it worthwhile, reviewers will be invited.
  4. If it is accepted, the authors of the publication are notified, provided with the accepted Letter, and given the opportunity to write a Rejoinder.
  5. The Letter is published together with the Rejoinder (not earlier).
  6. In some journals, both have even the same DOI so that interested readers are aware of both.
  • The letter was submitted on May 31st (#2). The editor invited me to review it on June 11th (#3) and notified me six (‼) minutes later that “we have already received sufficient commitments to review this manuscript”. #4 and #5 not followed. Was I notified that the letter was published on 9 September? Nope. I checked the website of the journal regulary…
    Is the letter [image] Open Access like our article? Nope. The [image] can’t afford that. Buy it for $ 39.95 or ask Wanjie for a copy.
Since Rejoinders are not possible in the AAPS J, we will submit a ‘Letter’ responding to a ’Letter’… :-D

Edit: Rejoinder submitted on 6 October.

Dif-tor heh smusma 🖖🏼 Довге життя Україна! [image]
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
BEQool
★    

2024-10-03 17:09
(67 d 12:12 ago)

@ Helmut
Posting: # 24211
Views: 1,482
 

 Group-by-Treatment Interaction: the final word?

Dear Helmut,

thanks for the article.
What should be used or what did you use as an error term (in the denominator) when testing group*treatment effect? MS Error (as for within-subject factors) or Subject(group×sequence) Error (as for between-subject factors)?

Regards
BEQool
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2024-10-03 17:33
(67 d 11:48 ago)

@ BEQool
Posting: # 24212
Views: 1,472
 

 ANOVA of Model 1

Hi BEQool,

❝ … what did you use as an error term (in the denominator) when testing group*treatment effect? MS Error (as for within-subject factors) or Subject(group×sequence) Error (as for between-subject factors)?


Here is the ANOVA of the first simulated study of our Scenario 1:

Response: log(Y)
                       Df      Sum Sq      Mean Sq F value   Pr(>F) 
group                   1 0.171667589 0.1716675892 2.12908 0.151630 
sequence                1 0.339810342 0.3398103421 4.21444 0.046057 *
treatment               1 0.349206911 0.3492069106 4.33098 0.043277 *
group:period            2 0.224598347 0.1122991734 1.39277 0.259120 
group:sequence          1 0.053695040 0.0536950403 0.66594 0.418865 
group:treatment         1 0.252244911 0.2522449107 3.12843 0.083870 .
group:sequence:subject 44 5.173880238 0.1175881872 1.45837 0.107359 
Residuals              44 3.547717653 0.0806299467                   
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

And then …

p.GxT[sim] <- anova(model1)[["group:treatment", "Pr(>F)"]]

 … which is 0.083870. Did we screw up?

Dif-tor heh smusma 🖖🏼 Довге життя Україна! [image]
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
BEQool
★    

2024-10-03 18:08
(67 d 11:13 ago)

@ Helmut
Posting: # 24214
Views: 1,508
 

 ANOVA of Model 1

❝ Here is the ANOVA of the first simulated study of our Scenario 1:

  Response: log(Y)

                         Df      Sum Sq      Mean Sq F value   Pr(>F)

  group                   1 0.171667589 0.1716675892 2.12908 0.151630 

  sequence                1 0.339810342 0.3398103421 4.21444 0.046057 *

  treatment               1 0.349206911 0.3492069106 4.33098 0.043277 *

  group:period            2 0.224598347 0.1122991734 1.39277 0.259120 

  group:sequence          1 0.053695040 0.0536950403 0.66594 0.418865 

  group:treatment         1 0.252244911 0.2522449107 3.12843 0.083870 .

  group:sequence:subject 44 5.173880238 0.1175881872 1.45837 0.107359 

  Residuals              44 3.547717653 0.0806299467       

  ---

  Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


From the table above if I am not mistaken I see that sequence effect was tested by using a Residuals (=MSE) as an error term. Shouldnt sequence effect be tested by using subject(sequence) or in this case subject(group*sequence) as the error term because it is a between subject factor?

I am wondering if group*treatment is also a between-subject factor and should therefore be tested by using subject(group*sequence) as the error term and not Residuals (similarly as sequence effect)?
Because based on this post and your reply (thanks!) sex, sequence, stage, group, site and also sex*treatment are between-subject factors and should therefore be tested by using subject(group*sequence) or subject(sequence) (if we dont have group in the model) as the error term. If sex*treatment is a between-subject factor, I assume group*treatment should also be?

❝ And then …

❝   p.GxT[sim] <- anova(model1)[["group:treatment", "Pr(>F)"]]

❝ … which is 0.083870. Did we screw up?


Probably not but based on my explanation above shouldnt then F value for group*treatment be 2.14516? So 0.2522449107 divided by 0.1175881872?

PS Isn't MS Residuals (=MSE) generally always pretty smaller than MS subject(sequence) or subject(sequence*group) in case of groups?
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2024-10-04 12:11
(66 d 17:10 ago)

@ BEQool
Posting: # 24215
Views: 1,448
 

 ANOVA of Model 1

Hi BEQool,

you have a point!

anova1 <- anova(model1)
num    <- as.numeric(anova1["group:treatment", c(3, 1)])
denom  <- as.numeric(anova1["group:sequence:subject", c(3, 1)])
cat(pf(num[1] / denom[1], num[2], denom[2], lower.tail = FALSE), "\n")

Gives for the first símulation above:

0.1501297


However, I’m puzzled because then p(G×T) of our meta study are not uniformly distributed any more. :confused:

[image][image]

Compare them to these plots.

Dif-tor heh smusma 🖖🏼 Довге життя Україна! [image]
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
mittyri
★★  

Russia,
2024-10-05 02:08
(66 d 03:13 ago)

(edited on 2024-10-05 22:59)
@ Helmut
Posting: # 24216
Views: 1,417
 

 Between vs Within

Hi Helmut & BEQool,

   anova1 <- anova(model1)

   num    <- as.numeric(anova1["group:treatment", c(3, 1)])

   denom  <- as.numeric(anova1["group:sequence:subject", c(3, 1)])


I really doubt that. In the article* referred there's a claim:

For the second and third comparisons, models 1 and 2 were used. Sex-by-formulation interactions were expressed by comparing the ratio of the test and reference geometric means for women with that for men. For sex-by-formulation interactions, observed ratio differences of greater than or equal to ±20 percentage points or statistically significant differences at P < .05 were used to identify interactions of interest.

It appears that the authors used the Residual Mean Squares (MS) as the denominator for the F-tests, rather than the Subject MS. This choice is not explicitly stated but seems likely based on the overall approach. Therefore, it's reasonable to assume that the interaction term Group*Formulation should also be tested against the Residual MS, not the Subject MS.


  • Chen M-L, Lee S-C, Ng M-J, Schuirmann DJ, Lesko LJ, Williams RL. Pharmacokinetic analysis of bioequivalence trials: Implications for sex-related issues in clinical pharmacology and biopharmaceutics. Clin Pharm Ther. 2000; 68(5): 510–21. doi:10.1067/mcp.2000.111184

Kind regards,
Mittyri
BEQool
★    

2024-10-06 22:06
(64 d 07:14 ago)

@ mittyri
Posting: # 24217
Views: 1,363
 

 Between vs Within

Hello Mittyri and Helmut,

❝ It appears that the authors used the Residual Mean Squares (MS) as the denominator for the F-tests, rather than the Subject MS. This choice is not explicitly stated but seems likely based on the overall approach. Therefore, it's reasonable to assume that the interaction term Group*Formulation should also be tested against the Residual MS, not the Subject MS.


So you would assume that Residual Mean Squares (MS) as the denominator was used because nothing is explicitly stated? If Subject MS was used instead as the denominator, you think that they would probably mention it?

Is there any source or literature what to use as denominator when assessing Group*Treatment interaction? I cant find anything relevant. Nothing is mentioned about it in the new ICH M13A guideline: The group x treatment interaction term should not be included in the model. However, applicants should evaluate potential for heterogeneity of treatment effect across groups and discuss its potential impact on the study data, e.g., by investigation of group x treatment interaction in a supportive analysis and calculation of descriptive statistics by group.

I am wondering what to use as a denominator when testing the Group*Treatment interaction and how to support this decision with literature references when writing a report and answering to regulatory agencies.

Additionally, what if the significance between the denominator used differ?

Regards
BEQool
mittyri
★★  

Russia,
2024-10-07 00:00
(64 d 05:21 ago)

(edited on 2024-10-07 00:31)
@ BEQool
Posting: # 24219
Views: 1,351
 

 Treatment-by-Group in PROC MIXED and PROC GLM?

Hi BEQool,

❝ So you would assume that Residual Mean Squares (MS) as the denominator was used because nothing is explicitly stated? If Subject MS was used instead as the denominator, you think that they would probably mention it?

yes.
The regulators are not trying to make our life easier. Yes, the model to be used is rarely presented. There are not so many exceptions (i.e. replicate designs for EMA or FDA).

❝ Is there any source or literature what to use as denominator when assessing Group*Treatment interaction? I cant find anything relevant. Nothing is mentioned about it in the new ICH M13A guideline

You are right. There's nothing. Note that when FDA published the code for 2*2*2 studies (a while ago), they did not mention in the text that a sequence should be tested against Subject MS, but just added a RANDOM statement in PROC GLM. Anyway, this decision had some reason.

In any sources with Group*Treatment interaction the denominator is not mentioned directly, but I think that without additional RANDOM statement (or ANOVA postprocessing) which SHOULD be mentioned if in use, the Residual MS is still valid. So you should prove why you modified your ANOVA to get F against Subject MS. The `famous` Sun et al. * paper states:

For each dataset, a Linear Mixed Model is fit with subgroup, sequence, subgroup-by-sequence, period nested within subgroup, treatment, treatment-by-subgroup as the fixed model and subject nested within subgroup and sequence as a random effect model (the same model can also be fit using the General Linear Model (GLM) with fixed effects only but with slight modification).


❝ Additionally, what if the significance between the denominator used differ?

yes, that's why it should be mentioned if the model/ANOVA was modified to use Subject MS


  • Wanjie Sun, Don Schuirmann & Stella Grosser (31 Oct 2022): Qualitative versus Quantitative Treatment-by-Subgroup Interaction in Equivalence Studies with Multiple Subgroups, Statistics in Biopharmaceutical Research, doi:10.1080/19466315.2022.2123385


PS: Are you equipped with SAS to check the ANOVA tables for this model (using some dataset without dropouts?) It would be interesting to see what PROC MIXED thinks about this interaction. From this paper I assume they should be the same as PROC GLM. But who knows! I could be wrong with my claim...

Kind regards,
Mittyri
BEQool
★    

2024-10-07 12:27
(63 d 16:54 ago)

@ mittyri
Posting: # 24220
Views: 1,293
 

 Treatment-by-Group in PROC MIXED and PROC GLM?

❝ […] So you should prove why you modified your ANOVA to get F against Subject MS. […]

I mean if the Group*Treatment is a between-subject factor, it should be, in my opinion, tested by using Subject MS as the denominator.

❝ The `famous` Sun et al. paper states:

❝ For each dataset, a Linear Mixed Model is fit with subgroup, sequence, subgroup-by-sequence, period nested within subgroup, treatment, treatment-by-subgroup as the fixed model and subject nested within subgroup and sequence as a random effect model (the same model can also be fit using the General Linear Model (GLM) with fixed effects only but with slight modification).

As already discussed, you think that if they had used MS Subject as the denominator, they would have probably written it here?

❝ PS: Are you equipped with SAS to check the ANOVA tables for this model (using some dataset without dropouts?) It would be interesting to see what PROC MIXED thinks about this interaction. From this paper I assume they should be the same as PROC GLM. But who knows! I could be wrong with my claim...

Yes I have access to SAS but I dont exactly understand what would you like to know? Which dataset are you talking about? Some random dataset or a specific one? And you are talking about testing Group*Treatment interaction with this dataset with both PROC MIXED and PROC GLM and to see if there is any difference?

Regards
BEQool
mittyri
★★  

Russia,
2024-10-07 15:11
(63 d 14:10 ago)

@ BEQool
Posting: # 24222
Views: 1,291
 

 Treatment-by-Group in PROC MIXED and PROC GLM?

Hi BEQool,

❝ ❝ […] So you should prove why you modified your ANOVA to get F against Subject MS. […]

❝ I mean if the Group*Treatment is a between-subject factor, it should be, in my opinion, tested by using Subject MS as the denominator.


<IMHO>
Subject MS (nested within subgroup and sequence) is typically used to test other between-subject factors, like sequence or subgroup effects, not treatment-related effects. The subject MS usually captures variability between subjects but within their respective subgroups and sequences. Treatment-by-subgroup MS should generally be tested against the residual MS, because this residual error captures the variability within subjects that remains after accounting for treatment and subgroup effects.
If the subject MS is used as the denominator, this could lead to inflated F-values because the subject MS reflects between-subject variability, which is not appropriate for testing interactions involving the treatment effect
</IMHO>

❝ As already discussed, you think that if they had used MS Subject as the denominator, they would have probably written it here?


yes

❝ Which dataset are you talking about? Some random dataset or a specific one?


No preferences in exact dataset, but it would be good to test the dataset with unequal groups length

❝ And you are talking about testing Group*Treatment interaction with this dataset with both PROC MIXED and PROC GLM and to see if there is any difference?


yes, if possible please

Kind regards,
Mittyri
BEQool
★    

2024-10-14 13:01
(56 d 16:19 ago)

@ mittyri
Posting: # 24227
Views: 1,037
 

 Treatment-by-Group in PROC MIXED and PROC GLM?

Hello Mittyri,

thank you for your comment regarding the denominator used for testing Group*Tretament effect, highly appreciated.

❝ ❝ Which dataset are you talking about? Some random dataset or a specific one?

❝ No preferences in exact dataset, but it would be good to test the dataset with unequal groups length

❝ ❝ And you are talking about testing Group*Treatment interaction with this dataset with both PROC MIXED and PROC GLM and to see if there is any difference?

❝ yes, if possible please


In SAS I performed ANOVA with dataset with unequal group sizes (and with subjects that completed both periods) with both PROC MIXED and PROC GLM to test Group*Treatment effect and the p-values for all PK parameters do not differ between PROC MIXED and PROC GLM. So you were right :ok:

BEQool
UA Flag
Activity
 Admin contact
23,335 posts in 4,901 threads, 1,665 registered users;
21 visitors (0 registered, 21 guests [including 6 identified bots]).
Forum time: 04:21 CET (Europe/Vienna)

Every man is fully satisfied that there is such a thing as truth,
or he would not ask any question.    Charles Sanders Peirce

The Bioequivalence and Bioavailability Forum is hosted by
BEBAC Ing. Helmut Schütz
HTML5