Bioequivalence and Bioavailability Forum • Rules acc. to Health Canada

Helmut
★★★

Vienna, Austria,
2019-02-05 16:04
(2256 d 08:01 ago)

Posting: # 19867
Views: 10,252

Rules acc. to Health Canada [Outliers]

Dear all,

I’m struggling in understanding HC’s rules. The current guidance states:

2.3.5 Outlier consideration
Comparative bioavailability studies are small studies compared to other clinical trials. One or two extreme values could have a large effect on the inference to be made from these small studies. The usual parametric assumptions and estimation are not robust against extreme values.

Specific procedures to identify and account for outliers should be pre-specified in the protocol. No more than 5% of the subjects may be considered to be outliers, unless there are 20 or fewer subjects, in which case only 1 subject may be removed. Any protocol for handling outliers should be followed before the results of the analysis are summarised into confidence intervals (i.e., regardless of whether results meet the standard, the outlier protocol should be followed).

The protocol for handling outliers should include the following.

The observation(s) should be identified by an outlier test. It is recommended that a simple outlier test, such as a studentised residual being greater than 3, be used.
The observation(s) should be outside the range of all the other observations regardless of formulation. In other words, the procedure should only identify observations which are very different from all others collected.
The subject in question should be identified as an outlier for all parameters, for either the test or reference product, upon which the bioequivalence decision is to be based. Parameters of interest are usually an AUC and C_max measure, but in some instances other parameters are required.

I’m fine with the introduction and the first two procedures. The third one gives me headaches. What is meant by “The subject in question should be identified as an outlier for all parameters”? It is not uncommon that one PK metric shows extreme values but not others. Is it really meant that in such a case the outlier cannot be excluded?

Background: I have a pilot study on my desk where 4 PK metrics are agreed upon with HC as primary (C_max, AUC_0–t, and two partial AUCs) for the pivotal study. One subject showed for the first pAUC a T/R-ratio of 6.993 (!) with a studentized residual of –7.198. Studentized residuals of the other metrics are fine. OK, it’s a pilot study. Would already pass BE with flying colors for 3 metrics but the upper 90% CI for the first pAUC is 156.4% (CV 50.1%). After exluding the subject the CV would decrease to 22.1%. Do I understand the guidance correctly: I’m not allowed to exclude the pAUC of the outlying subject cause the other metrics are fine?
:confused:

in writing the protocol of the pivotal study.

—
Dif-tor heh smusma 🖖🏼 Довге життя Україна!
Helmut Schütz

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

ElMaestro
★★★

Denmark,
2019-02-05 16:30
(2256 d 07:35 ago)

@ Helmut
Posting: # 19868
Views: 9,040

Rules acc. to Health Canada

Post reply

Hi Helmut,

yes it was meant to indicate that if one metric is outlier then the others should be as well before exclusion is appropriate. However, I don't think they are always enforcing it that way. After all, rate is not the same as extent and vice versa.

But does it make a big difference? Exclusion changes little in your case since it was a pilot and you will need to consider the sample size and relevance of the CV and PE sampled in the pilot when planning the pivotal in any case.

If Cmax and AUC were more or less normal for the subject, then in all likelihood she munched the medication as directed to and what you saw is populations , products and random happenings at work.

—
Pass or fail!
ElMaestro

Helmut
★★★

Vienna, Austria,
2019-02-05 17:24
(2256 d 06:41 ago)

@ ElMaestro
Posting: # 19872
Views: 9,014

Rules acc. to Health Canada

Post reply

Hi ElMaestro,

THX for your nice words!

❝ yes it was meant to indicate that [ :blahblah: ].

Yep, that’s what I assumed as well.

❝ However, I don't think they are always enforcing it that way.

Shall I cross fingers?

❝ After all, rate is not the same as extent and vice versa.

Xactly!

❝ But does it make a big difference? Exclusion changes little in your case since it was a pilot …

No problems with this one. I’m worried what will happen in the pivotal study.

❝ … and you will need to consider the sample size and relevance of the CV and PE sampled in the pilot when planning the pivotal in any case.

But that’s the point! This one subject drives the PE to 110% and with a CV of ~50% I end up with ~200 subjects in a 2×2. OK, for HC (and even for the EMA cause it’s a MR) I could apply reference-scaling. 28 subjects in a 2×4. Fine, but the pivotal is a little bit tricky (5 arms already, don’t ask me why). Hence, I don’t want to go there.

❝ If Cmax and AUC were more or less normal for the subject, …

T/R of C_max 1.115 and of AUC_0–t 1.036…

❝ … then in all likelihood she munched the medication as directed to and what you saw is populations, products and random happenings at work.

Oh dear! There were formulation changes (plural!) which were acceptable to be supported by dissolution similarity on this side of the pond. AFAIK, this stuff is marketed in ~50 countries. So far, so good. Rules are different. For one of the changes in CAN one would need a BE study. Hence, a lot of stuff to do. The original application was a hybrid (two clinical studies). In order to bridge to the clinical stuff HC wants a BE study of the current formulation against the “old” one.
When looking at the individual curves I would say that the current formulation is closer to the originally aimed target profile. In this subject the old formulation looks bad in the first couple of hours. That’s my problem.

—
Dif-tor heh smusma 🖖🏼 Довге життя Україна!
Helmut Schütz

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

ElMaestro
★★★

Denmark,
2019-02-05 18:33
(2256 d 05:32 ago)

@ Helmut
Posting: # 19873
Views: 8,994

Rules acc. to Health Canada

Post reply

Hi Hötzi,

❝ Shall I cross fingers?

I am inclined to just write what the guideline says. Cross fingers, light 15 candles and say a lot of Ave Marias. Always works.

❝ But that’s the point! This one subject drives the PE to 110% and with a CV of ~50% I end up with ~200 subjects in a 2×2. OK, for HC (and even for the EMA cause it’s a MR) I could apply reference-scaling. 28 subjects in a 2×4. Fine, but the pivotal is a little bit tricky (5 arms already, don’t ask me why). Hence, I don’t want to go there.

I am delighted I am not having the responsibility for taking the decision about pivotal design and sample size given that info :-D

On the rare occasion when I encounter clients who want 5 arms then usually those are innovators who are entering the game of generics or BE and who, one way or another, think bioequivalence is just a very simplified kind of innovator thinking. The attitude to it is quite leaned back. It results in those funny designs which don't correspond to the philosophy of guidelines and where the complications far, far exceed the advantages.
It is God's gift to my blood pressure that I don't get involved in those developments very often. :-)

—
Pass or fail!
ElMaestro

Ohlbe
★★★

France,
2019-02-05 16:54
(2256 d 07:11 ago)

@ Helmut
Posting: # 19870
Views: 9,065

Rules acc. to Health Canada

Post reply

Dear Helmut,

❝ Do I understand the guidance correctly: I’m not allowed to exclude the pAUC of the outlying subject cause the other metrics are fine?

Interesting. Well, pAUC was added to the guidance in June 2018, but the section on outliers was already there. Not sure that they considered all implications when they revised the guidance. Maybe it would be worth contacting Health Canada to ask for a confirmation ? They may have only had Cmax and AUCt in mind when this paragraph was written ?

—
Regards
Ohlbe

Helmut
★★★

Vienna, Austria,
2019-02-05 18:52
(2256 d 05:13 ago)

@ Ohlbe
Posting: # 19875
Views: 8,997

Rules acc. to Health Canada

Post reply

Hi Ohlbe,

❝ […] pAUC was added to the guidance in June 2018, but the section on outliers was already there. Not sure that they considered all implications when they revised the guidance. Maybe it would be worth contacting Health Canada to ask for a confirmation ? They may have only had Cmax and AUCt in mind when this paragraph was written ?

Maybe. This part of (3) is funny:

Parameters of interest are usually an AUC and C_max measure, but in some instances other parameters are required.

Others? Wow! I guess they mean additional ones. The only studies I ever have done without AUC and C_max were for an antibiotic where primary were a set of Occupany times. ;-)

And yes, we are in constant contact with HC. This part slipped through my attention.

—
Dif-tor heh smusma 🖖🏼 Довге життя Україна!
Helmut Schütz

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

mittyri
★★

Russia,
2019-02-05 16:58
(2256 d 07:07 ago)

@ Helmut
Posting: # 19871
Views: 9,088

for either the test or reference product?

Post reply

Hi Helmut,

maybe I'm only one who is confused with that statement, but:
3. The subject in question should be identified as an outlier for all parameters, for either the test or reference product, upon which the bioequivalence decision is to be based.

Either? Sorry for my poor grammar knowledge but 'either' is a kind of choice, right? So does it mean we need to find out the outliers using the datasets splitted by formulation, not for the ratio T/R? :confused:

—
Kind regards,
Mittyri

Helmut
★★★

Vienna, Austria,
2019-02-05 18:42
(2256 d 05:23 ago)

@ mittyri
Posting: # 19874
Views: 9,067

Discordant outliers

Post reply

Hi Mittyri,

❝ maybe I'm only one who is confused with that statement, but:

❝ 3. The subject in question should be identified as an outlier for all parameters, for either the test or reference product, upon which the bioequivalence decision is to be based.

❝

❝ Either? Sorry for my poor grammar knowledge but 'either' is a kind of choice, right?

In the combination either – or, absolutely. If you drop the or it is a synonym for each (element of a set).
Either A or B: A ∨ B
Either of A, B, C: A ∧ B ∧ C

I guess HC means that it doesn’t matter whether an aberrant response is seen after test or reference. Agrees with ICH E9 Section 5.3:

Any outlier procedure set out in the protocol or the statistical analysis plan should be such as not to favour any treatment group a priori.

❝ So does it mean we need to find out the outliers using the datasets splitted by formulation, not for the ratio T/R? :confused:

Good point. If we split the dataset, we can’t run the model, right? Studentized residuals? Nada.
I guess HC talks about discordant outliers. In other words, it doesn’t matter whether the aberrant response is observed for the test or reference. Like it.

—
Dif-tor heh smusma 🖖🏼 Довге життя Україна!
Helmut Schütz

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

Shuanghe
★★

Spain,
2019-02-06 12:48
(2255 d 11:17 ago)

@ Helmut
Posting: # 19880
Views: 8,912

Rules acc. to Health Canada

Post reply

Dear all,

On slightly related issue. Canadian guideline says max 5% outliers can be removed and one criterion for outlier test is studentised residual > 3.

I had an incidence in a pilot study where there are more than 5% subjects having studentised residual > 3. Since it's a pilot, it doesn't matter too much but in the protocols of pivotal studies I made it very clear that if this happens, the 5% subjects to be removed will be chosen by the magnitude of the residuals: only remove those with highest residual (in absolute value, e.g., those with residual -7 will be removed before those with residual 5). For the rest of those subjects, even though they are identified as outlier, they will be included in the BE evaluation, in accordance with guidance's 5% rule. Unfortunately (or fortunately), it never happens to my pivotal studies so I have no idea if Canadian agency is really ok with this.

Do any of you have such experience?

—
All the best,
Shuanghe

ElMaestro
★★★

Denmark,
2019-02-06 13:20
(2255 d 10:45 ago)

@ Shuanghe
Posting: # 19881
Views: 8,947

Rules acc. to Health Canada

Post reply

Hi Shuanghe,

wouldn't you remove subject (sing.) with the highest absolute magnitude of studentised residual first, and then re-residual them all (having the first subject removed) before considering removal of the next?
I would do that, to perpetuate the thinking behind removal of calibrators that is widely used in bioanalytics.
It is the same kind of apples and pears :-D

—
Pass or fail!
ElMaestro

mittyri ★★ Russia, 2019-02-06 15:04 (2255 d 09:01 ago) @ ElMaestro Posting: # 19882 Views: 8,825	ESD? Post reply
	Hi ElMaestro, ❝ wouldn't you remove subject (sing.) with the highest absolute magnitude of studentised residual first, and then re-residual them all (having the first subject removed) before considering removal of the next? do you mean ESD? — Kind regards, Mittyri

ElMaestro
★★★

Denmark,
2019-02-06 19:12
(2255 d 04:53 ago)

@ mittyri
Posting: # 19885
Views: 8,925

ESD?

Post reply

Hi Mittyri,

❝ do you mean ESD?

I mean more or less:

1. Fit the model on the dataset
2. Evaluate if there is an outlier (or more than one) by whatever criterion such as stud.res. >3
3a. If there is no outlier proceed to 4.
3b. If there is an outlier, remove it. This defines a new dataset. Cycle back to step 1. In the case of multiple aberrant values noted under step 2, focus only on the one whose magnitude is highest, eg. the highest absolute stud. res.
4. Do stats via the fitted model.

I don't know if this is ESD. It is an abbreviation I am not familiar with and I am on my way out, so not too much time to investigate the link. But let me guess:
Emily Saw Dennis? Every Skunk Dies? Eagles Soar Deliriously? Eat Sh!t and Die? Evelyn Sauntered Diagonally?

—
Pass or fail!
ElMaestro

zizou
★

Plzeň, Czech Republic,
2019-02-06 21:52
(2255 d 02:13 ago)

@ Helmut
Posting: # 19886
Views: 8,960

Rules acc. to Health Canada

Post reply

Hello everybody and nobody.

I'm just thinking about another slightly related issue.

Assume that there is an outlier in the study for Health Canada.
If the outlier wasn't removed, we would get GMR₁, intra-subject CV₁ and corresponding 90% CI.
However the outlier is removed according to the study protocol and the results are different:

GMR T/R can be higher or lower than GMR₁ randomly.
intra-subject CV will be always lower than intra-subject CV₁.

So when outlier is removed as planned, my question is: "Will be the patient's risk (Type I Error) still lower or equal to 5%?"

Assume the border case with true GMR 0.8 and intra-subject CV e.g. 30%. The TIE (in the border case equal to power) in standard 2x2 BE is 5%. However I am not sure if it is the same with exclusion of outlier. So do we get some bonus power as intra-subject CV will be always lower after exclusion? I just imagine the simulation - individual simulated cases with:

observed GMR lower or equal to 0.8 - the same as standard 2x2, i.e. not bioequivalent cases.
observed GMR is higher than 0.8 - as observed intra-subject CV is lower in cases with the exclusion of outlier(s) - there will be slightly more bioequivalent cases than in standard 2x2 without exclusion of outlier(s).

So I would expect the TIE slightly over 5%. (Wenn ich mich nicht irre.)

Maybe someone could try to simulate it - TIE with the exclusion of outlier(s) for BE in 2x2.
I am not familiar with outliers and complicated simulations - simulate data (true GMR 0.8 and intra-subject CV 30%), look for outliers and exclude them (it could end with lowering the intra-subject CV), calculate 90% CI, and do it million times. The result will be probably close to 5%, but still higher?

Best regards,
zizou

Helmut
★★★

Vienna, Austria,
2019-02-07 11:37
(2254 d 12:28 ago)

@ zizou
Posting: # 19887
Views: 8,730

Great post!

Post reply

Hi Zizou,

at a first look your arguments are convincing indeed.

A basic assumption in BE is that what we observe in the study is an unbiased sample of the population, i.e., the true (but unknown) distributions of the sample and the population are identical. Only then we can extrapolate the study’s result to the patient population (BE as a substitute for therapeutic equivalence). As stated in HC’s guidance parametric methods are not robust against extreme values. My example:

n method PE 90% CI CV_intra CV_inter 12 LME 110.15% 77.60–156.36% 50.1% NA (negative variance component)
12 ANOVA 110.15% 77.60–156.36% 50.1% NA 11 LME 92.45% 78.34–109.11% 21.3% 22.2% 12 nonparametric 98.95% 76.24–130.06% NA NA (note: 91.78% CI)

Observations:

The mixed effects model (according to the guidance) converged but with a negative variance for the random effect. Tweaks weren’t successful (singularity tolerance ↓, convergence criterion ↓, maximum number of iterations ↑).
Excluding the outlier I got a pretty narrow CI (which is similar to the other PK metrics). However, the one in the nonparametric method of the full dataset is much wider. Is the distribution of n=11 really close to the true one?
In a strict sense the nonparametric method gives only a correct measurement of shift if distributions are similar. Duno whether this is the case here. Haven’t tried the Brunner-Munzel test* yet (which doesn’t require this assumption).

Time allowing (ha-ha) I will try some sim’s where I replace one value with the true x±6σ. Asymptotics are generally fine if n≥30. Hence your suggestion for CV 30% makes sense (GMR 0.95, power 80% → n 40). But I will also try one with a lower CV of 15% (n 12).

Brunner E, Munzel U. (2000). The Nonparametric Behrens-Fisher Problem: Asymptotic Theory and a Small Sample Approximation. Biometr. J. 2000;42(1):17–25. doi:10.1002/(SICI)1521-4036(200001)42:1<17::AID-BIMJ17>3.0.CO;2-U.

—
Dif-tor heh smusma 🖖🏼 Довге життя Україна!
Helmut Schütz

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes