Bioequivalence and Bioavailability Forum 13:50 CET

Main page Policy/Terms of Use Abbreviations Latest Posts

 Log in |  Register |  Search

Lucas
Regular

Brazil,
2014-02-19 14:27

Posting: # 12447
Views: 11,536
 

 Qualitative analysis of the ANOVA residuals [General Sta­tis­tics]

Hello everybody.

We've been asked to perform a qualitative analysis of the residuals of the ANOVA. Now, I don't know what that analysis is supposed to acomplish, so I don't know what is supposed to be done. Do you guys have any experience with that? Accept my apologies in advance if that is too much of beginner's question.

Thank you.
Helmut
Hero
avatar
Homepage
Vienna, Austria,
2014-02-19 15:22

@ Lucas
Posting: # 12450
Views: 10,917
 

 Qualitative analysis of the ANOVA residuals

Hi Lucas,

» We've been asked to perform a qualitative analysis of the residuals of the ANOVA.

ANVISA?

» Now, I don't know what that analysis is supposed to acomplish, so I don't know what is supposed to be done.

IMHO, assessing the studentized residuals makes sense. See also Canada’s Guidance, Section 2.3.4.
  1. From the model’s raw residuals calculate the standardized residuals (aka internally studentized residuals).
  2. From them calculate the studentized residual (aka deleted studentized residual, extrenally studentized residual).
Both are directly accessible in SAS and R – and with a little bit of struggle – in Phoenix/WinNonlin. I would suggest to come up with a box-plot (optionally overlayed with jittered values) and ±3 as an optical reference. Looks like this:

[image]

Cheers,
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. ☼
Science Quotes
Lucas
Regular

Brazil,
2014-02-19 15:58

@ Helmut
Posting: # 12453
Views: 10,804
 

 Qualitative analysis of the ANOVA residuals

Hi Helmut! Thanks for the fast response, as always.

» ANVISA?

No, the sponsor wants us to perform that. The study would be submitted at ANVISA, but I think they want a study that can be (if they ever need) submitted at other agencys. I just love how they ask us that kind of analysis without even explaining their objective with that. So I thought that you guys had experience with that being required in other countries. The sponsor is from England, if that helps.

» IMHO, assessing the studentized residuals makes sense. See also Canada’s Guidance, Section 2.3.4.

Doesn't seem to be a criteria for saying if the results are good or not, am I right?

» [...] and with a little bit of struggle – in Phoenix/WinNonlin.

Can u tell me how it can be done at Phoenix?

Thanks.
Helmut
Hero
avatar
Homepage
Vienna, Austria,
2014-02-19 16:25

@ Lucas
Posting: # 12456
Views: 10,804
 

 Qualitative analysis of the ANOVA residuals

Hi Lucas,

» […] So I thought that you guys had experience with that being required in other countries.

Limited experience. Once I had to deal with studies on a HVD (pre-scaling age) where the spon­sor – based on previous experiences – stated in the protocols that based on assessing the resid­uals (box­plots and Shapiro-Wilk test) in case of non-normality the study will be evaluated by a non­parametric method. The Dutch agency didn’t like that. In a hearing in The Hague the told me that “We in The Netherlands don’t believe in nonparametric statistics”. ;-)

» Doesn't seem to be a criteria for saying if the results are good or not, am I right?

In Canada you may exclude outliers if you have stated criteria in the protocol.

» » [...] and with a little bit of struggle – in Phoenix/WinNonlin.
» Can u tell me how it can be done at Phoenix?

I’m currently struggling with some PopPK/PD-models which send my CPUs’ load to 80%. I don’t want to start concurrently another PHX-session because it would exceed my RAM and Phoenix would start swapping memory to disk. If I don’t come up with something in the next two weeks, remind me. Some hints in my whitepaper (Section 3.6) about EMA’s ABEL.

Cheers,
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. ☼
Science Quotes
Lucas
Regular

Brazil,
2014-02-19 17:20

@ Helmut
Posting: # 12457
Views: 10,754
 

 Qualitative analysis of the ANOVA resid­uals

» I’m currently struggling with some PopPK/PD-models which send my CPUs’ load to 80%. I don’t want to start concurrently another PHX-session because it would exceed my RAM and Phoenix would start swapping memory to disk. If I don’t come up with something in the next two weeks, remind me. Some hints in my whitepaper (Section 3.6) about EMA’s ABEL.

Thanks for the hints. If I can not perform that by reading the whitepaper I'll come back to you later.

Best regards.
Lucas
Regular

Brazil,
2014-02-20 19:07

@ Helmut
Posting: # 12464
Views: 10,587
 

 Qualitative analysis of the ANOVA resid­u­als

Hi Helmut and ElMaestro!

So, I got an answer from the sponsor about what would that qualitative analysis be. Both your's and ElMaestro's answers were right! This is how it would look like according to them:

[image]

Now my problem is how I would do that in Phoenix. I haven't got the time to read your whitepaper but I will soon, I only want to show you guys how this analysis would look like. Maybe it helps someone of the forum in the future.

Thank u.
Helmut
Hero
avatar
Homepage
Vienna, Austria,
2014-02-20 20:11

@ Lucas
Posting: # 12465
Views: 10,841
 

 First hints for Phoenix

Hi Lucas,

» This is how it would look like according to them: […]

Interesting! I didn’t count, but ~200 subjects is not common in BE studies. It will not look that nice in the real world. The x-axis of the first plot is strange (negative means – I beg your pardon?). The distribution doesn’t look normal but skewed to the left* and the Q-Q plot looks extremely ugly. That’s actually a great example which EDA would suggest to use another trans­formation (e.g., ℯx or xn, where n>1).
However, let’s see what you can get in PHX (I numbered the panels in your plot):
  1. Plotting > XY-Plot (though for the abscissa I would either use the subject’s ID or the pre­dicted model values).
  2. Plotting > Histogram Show relative Frequency. In the current version (6.3) it is not possible to overlay a normal distribution, maybe in the next release?
  3. Plotting > QQ Plot.
  4. Upper part = trivial. Lower part = strange. In PHX the line in the Q-Q plot is fitted auto­ma­tic­ally – I don’t think it is possible to extract the parameters and AIC, BIC.
    BTW, these values are only useful in comparing different models. I cannot imagine what might its purpose here. AICC is the AIC corrected for small sample sizes. It proves that the statis­ti­ci­an is a “push-the-button-type” who does not know what he is doing. Of course you could fit the standard quantiles vs. stud. resid’s by PHX’ linear model.
If the sponsor insists on a similar layout, you have to leave PHX because different types of plots arranged in an array (aka trelis/lattice plots) are not supported. Either export relevant data from PHX and continue the analysis in R or generate the plots in PHX and paste them into a 4-cell table in your preferred text-processor.


  • AUC and Cmax have been shown to follow a log-normal distribution – which is skewed to the right. However, in order to apply parametric methods the residuals (which are esti­mates of the residual error) must be normal – not the original values. The log-trans­­formation is more a convention than justified. I have never seen values skewed to the left. Which kind of stuff is this?

Cheers,
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. ☼
Science Quotes
Helmut
Hero
avatar
Homepage
Vienna, Austria,
2014-02-21 19:25

@ Lucas
Posting: # 12472
Views: 10,649
 

 Phoenix: resid­u­als

Hi Lucas,

in the following a setup for PHX6.3+ (Clayton/Leslie’s data given in Chow/Liu’s table 6.9.1; validated against table 8.2.3); sequences coded as TR|RT. It seems complicated first, but works also for unba­lanced sequences. This is not PHX’ standard model, but all effects fixed (like for EMA and in SAS Proc GLM). You have to code it only once and can use it as a template in other studies.
  1. Setup the BE wizard for ln-transformed data.
    Fixed effects: subject+period+treatment+sequence+subject(sequence),
    delete Random1: subject(sequence)
  2. Execute and navigate to Output Data > Residuals > Send To > Data > Data Wizard > Add Filter
    • Exclude everything except Dependent, Subject, Predicted, Residual (i.e., keep them mapped to None)
    • Built In > Add > Action: Include | Find: RT | Search Area Column: Sequence > OK
    • Built In > Add > Action: Include | Find: 1 | Search Area Column: Period > OK
    • Execute and rename to RT Filter
  3. Final Results > Result > Send To > Data > Split Worksheet
    • Map Sequence to Sort
    • Create unique values worksheets (uncheck the box!)
    • Execute and rename to RT Count
  4. Navigate back to Bioequivalence > Output Data > Final Variance Parameters > Send To > Data > Join Worksheets
    • Map Dependent to Sort, Estimate to Source Column, leave Parameter mapped to None
    • Map Worksheet 2 > RT Filter.Result
    • Map Dependent to Sort and all variables to Source Column
    • Execute and rename to RT MS intra
  5. Final Results > Result > Send To > Data > Join Worksheets
    • Map Sequence to Sort and all other variables to Source Column
    • Map Worksheet 2 > RT Count.Result
    • Map Sequence to Sort and Count to Source Column
    • Execute and rename to RT data
  6. Final Results > Result > Send To > Data > Data Wizard > Add Transformation
    • Transformation Type: Custom | Transformation: Custom Function | New Column Name: StudResIntra |
      Formula: Residual/sqrt(Estimate*(Count-1)/(2*Count))
    • Add Filter > Exclude Residual, Estimate, Count
    • Execute and rename to RT Stud Res intra
  7. Repeat steps #2 – #6 for sequence TR. You should end up with TR Filter, TR Count, TR MS intra, TR data, TR Stud Res intra.
  8. Navigate back to RT Stud Res intra > Final Results > Result > Send To > Data > Append Worksheets
    • Map all variables to Source Column
    • Map Worksheet 2 > TR Stud Res intra.Result
    • Map all variables to Source Column
    • Execute and rename to Stud Res intra
Using the standard plotting functions (see above), we get:

[image] [image]

[image]   [image]

[image]

In Phoenix Pre-Release V1.4 (Build 6.4.0.511, 2013-12-20) still no overlays in box plots and his­to­grams. :lookaround: Maybe I will post some R-code later.


Eidt: I have to correct myself: In Phoenix (all versions) these are standardized (sometimes call internally studentized) residuals, not studentized (externally studentized) residuals.

Cheers,
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. ☼
Science Quotes
BRN
Junior

Turkey,
2014-03-04 15:15

@ Helmut
Posting: # 12554
Views: 10,235
 

 Phoenix: resid­u­als

Dear Helmut,

I'm trying to run this process on WNL 6.3 but I'm getting confused in step 3 where I need to map sequence but there's no sequence since I exclude it in the previous step (2) because it's mentioned:

» Exclude everything except Dependent, Subject, Predicted, Residual (i.e., keep them mapped to None).

Could you please kindly suggest?

Regards,
BRN
Helmut
Hero
avatar
Homepage
Vienna, Austria,
2014-03-04 16:52

@ BRN
Posting: # 12555
Views: 10,244
 

 Phoenix: resid­u­als

Hi BRN,

» […] I'm getting confused in step 3 where I need to map sequence but there's no sequence since I exclude it in the previous step (2) because it's mentioned:
» » Exclude everything except Dependent, Subject, Predicted, Residual (i.e., keep them mapped to None).

In Step 2 you have to keep Sequence mapped to None. I made a stupid error – sorry for the confusion caused.

Cheers,
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. ☼
Science Quotes
Rosy
Junior

Mexico,
2018-10-15 07:03

@ Helmut
Posting: # 19442
Views: 2,696
 

 Phoenix: resid­u­als

Dear Helmut:

I do studentized residual test with R for the outliers detection, but now my regulation prohibid to use R. Can i do with phoenix 7 or another statistical software like ncss. than u


Edit: Full quote removed. Please delete everything from the text of the original poster which is not necessary in understanding your answer; see also this post #5[Helmut]
Helmut
Hero
avatar
Homepage
Vienna, Austria,
2018-10-15 11:43

@ Rosy
Posting: # 19443
Views: 2,651
 

 Software validation

Hi Rosy,

» I do studentized residual test with R for the outliers detection, but now my regulation prohibid to use R.

Bizarre. Consider presenting the document

R: Regulatory Compliance and Validation Issues
A Guidance Document for the Use of R in Regulated Clinical
Trial Environments

and to ask the agency why it believes (!) that commercial software (SAS, Phoenix, …) is “better” than open source. Any software has to be validated* – which is easy for base-R.

» Can i do with phoenix 7 or another statistical software like ncss.

In Phoenix you get only standardized residuals (aka internally studentized residuals). For the setup see above. In SAS both internally and externally studentized residuals are possible. Haven’t used NCSS for 15+ years – no idea.


* Examples where a commercial software screwed up:
  1. Schütz H, Labes D, Fuglsang A. Reference Datasets for 2-Treatment, 2-Sequence, 2-Period Bioequivalence Studies. AAPS J. 2014;16(6):1292–7. doi:10.1208/s12248-014-9661-0. [image] free view-only version.
  2. Fuglsang A, Schütz H, Labes D. Reference Datasets for Bioequivalence Trials in a Two-Group Parallel Design. AAPS J. 2015;17(2):400–4. doi:10.1208/s12248-014-9704-6. [image] free view-only version.

Cheers,
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. ☼
Science Quotes
Rosy
Junior

Mexico,
2018-10-18 00:38

@ Helmut
Posting: # 19462
Views: 2,459
 

 Software validation

Dear Helmut

I know, what's wrong with my regulation, anyway... that's because I'm Mexican (if you remember Guillermo del Toro speach) LOL

I will try to follow you directions on phoenix, but someone tell me that it's valid to make a linear regression between Ln_observed and predicted from "Residuals" output from Phoenix and obtain the Student residuals to find outliers, is that right?
Helmut
Hero
avatar
Homepage
Vienna, Austria,
2018-10-18 17:14

@ Rosy
Posting: # 19470
Views: 2,427
 

 Avoid raw residuals

Hi Rosy,

» I know, what's wrong with my regulation, anyway... that's because I'm Mexican (if you remember Guillermo del Toro speach) LOL

This part?

» I will try to follow you directions on phoenix, …

Good luck.

» … but someone tell me that it's valid to make a linear regression between Ln_observed and predicted from "Residuals" output from Phoenix

What? That’s not interesting because it is rather difficult to “see” anything useful.*
You could plot the residuals (y) vs. Ln_predicted (x). Cave: In a 2×2 crossover for every subject you have two residuals with opposite signs. You need only one of them (commonly the first period is used). However, raw residuals are not very useful. Though you get an idea about trends (if they increase/decrease or show a strange shape the model is wrong) or homo-/heteroscedasticity (funnel shape) their magnitude is not telling. That’s why you should standardize/studentize them.

» … and obtain the Student residuals to find outliers, is that right?

Yep. Though I would not call them outliers (regulators don’t like it) but “aberrant values”. ;-)


  • That’s why in comparing two bioanalytical methods nobody plots method 1 / method 2 but uses the Bland-Altman plot.

Cheers,
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. ☼
Science Quotes
Rosy
Junior

Mexico,
2018-10-18 18:05
(edited by mittyri on 2018-10-18 18:23)

@ Helmut
Posting: # 19471
Views: 2,428
 

 Avoid raw residuals

Dear Helmut,


This video, i mean this became a meme for Mexicans, although everything it's black (wrong) you will see the light... Mexican Humor LOL


» Good luck.

Thank u, but i have trouble, i finish with 4 residuals for each patients (from final variance parameters) i dont know how analyze, because in R appears a table with intra and inter subject residuals for each subject, if i wanna make the same table which values i have to pick up.


» ...That’s why you should standardize/studentize them.

yep, i mean, i took "Ln_Observed" and "predicted" from Phoenix, then in ncss i run lineal regression and from that i do a regression diagnosis to obtain studentized residual, Do i have to do this with every sequence in every period (whole data) or just for the 1st period, of course the objetive it's to find aberrant values.... i know maybe this no make any sense but i have to comply my regulation


thank u for your patience, i'm new in bioequivalence but i hope to learn so much



Edit: youtube link is corrected. [Mittyri]
Helmut
Hero
avatar
Homepage
Vienna, Austria,
2014-02-23 02:20

@ Lucas
Posting: # 12473
Views: 10,572
 

 Phoenix → R

Hi Lucas,

export the stuff to R and some quick/dirty code:
In the Phoenix-project navigate to Stud Res intra > Output Data > Result right-click > Export
Filename: StudResIntra
Filetype: Text (Tab delimied) (*.txt)
Save

The default location is the desktop (which conveniently is also the default import site in R). Start R and paste this code into the console:

res <- read.table("StudResIntra.txt", header=TRUE, sep="\t")
attach(res)
ResLim <- c(-max(abs(StudResIntra))*1.1, max(abs(StudResIntra))*1.1)
PreLim <- c(min(abs(Predicted))*1.1, max(abs(Predicted))*1.1)
SubLim <- c(min(abs(Subject))*1.1, max(abs(Subject))*1.1)
TR <- subset(res, Sequence == "TR")
RT <- subset(res, Sequence == "RT")
RL <- function() {
  abline(h=c(0, -2, 2, -3, 3), lty=c(1, rep(2, 4)),
    col=c(rep("black", 2), rep("red", 2)))
}
lab <- "Studentized residual"
split.screen(c(3, 2))
screen(1)
plot(0, 0, type="n", xlim=PreLim, ylim=ResLim, xlab="Predicted", ylab=lab,
  main="Scatterplot 1", cex.main=0.9, las=1)
RL()
points(TR$Predicted, TR$StudResIntra, pch=23, col="red", bg="#ffdddd")
points(RT$Predicted, RT$StudResIntra, pch=22, col="darkgreen", bg="#ddffdd")
screen(2)
plot(0, 0, type="n", xlim=SubLim, ylim=ResLim, xlab="Subject", ylab=lab,
  main="Scatterplot 2", cex.main=0.9, las=1)
RL()
points(TR$Subject, TR$StudResIntra, pch=23, col="red", bg="#ffdddd")
points(RT$Subject, RT$StudResIntra, pch=22, col="darkgreen", bg="#ddffdd")
screen(3)
HisLim <- c(0, max(hist(StudResIntra, breaks="FD", plot=FALSE)$density)*1.1)
hist(StudResIntra, breaks="FD", freq=FALSE, xlim=ResLim, ylim=HisLim, xlab=lab,
  ylab="Frequency", main="Histogram", cex.main=0.9, las=1, border="blue",
  col="#ddddff")
curve(dnorm(x, mean=mean(StudResIntra), sd=sd(StudResIntra)), add=TRUE)
box()
screen(4)
qqnorm(StudResIntra, xlim=ResLim, ylim=ResLim, xlab="Theoretical quantiles",
  ylab=lab, main="Normal Q-Q plot", cex.main=0.9, pch=21, col="blue", bg="#ddddff",
  las=1, plot.it=TRUE)
qqline(StudResIntra, datax=F, distribution=qnorm)
screen(5)
boxplot(StudResIntra, xlim=c(0.5, 2.5), ylim=ResLim, ylab=lab, main="Boxplot",
  cex.main=0.9, las=1, at=1, border="blue", col="#ddddff")
RL()
points(x=jitter(rep(2, length(StudResIntra)), factor=4), StudResIntra,
  pch=21, col="blue", bg="#ddddff")
close.screen(all = TRUE)
detach(res)


[image]

Good luck! If you have a Connect license you even don’t have to leave PHX.

Cheers,
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. ☼
Science Quotes
Lucas
Regular

Brazil,
2014-02-25 19:28

@ Helmut
Posting: # 12497
Views: 10,393
 

 Phoenix → R

Wow!
Thanks for the help Helmut!

I got tied up with other projects and only saw your responses now. Sorry about that. Thank god I got an e-mail from the sponsor a few hours ago saying that we no longer have to perform that analysis. :-D
But these hints will be very helpful for us. Thank you very much for your helpfulness!

Best regards.
ElMaestro
Hero

Denmark,
2014-02-19 15:32

@ Lucas
Posting: # 12451
Views: 10,711
 

 Qualitative analysis of the ANOVA residuals

Hi,

it really depends on what your agency wants with that qualitative analysis of residuals.

I would do a histogram of residuals, possibly a histogram of within-subject differences.
Ideally that would amount to something that looks like a normal distribution, but realistically with a low number of subjects it may just look like the teeth of someone who pissed Mike Tyson seriously off. Looking for outliers (looking in this regards means looking, and not quantifying their potential presence) may be another aspect where this could come in handy.

if (3) 4

Best regards,
ElMaestro

"(...) targeted cancer therapies will benefit fewer than 2 percent of the cancer patients they’re aimed at. That reality is often lost on consumers, who are being fed a steady diet of winning anecdotes about miracle cures." New York Times (ed.), June 9, 2018.
Helmut
Hero
avatar
Homepage
Vienna, Austria,
2014-02-19 16:06

@ ElMaestro
Posting: # 12454
Views: 10,771
 

 Qualitative analysis of the ANOVA residuals

Hi ElMaestro,

» Ideally that would amount to something that looks like a normal distribution, but realistically with a low number of subjects it may just look like the teeth of someone who pissed Mike Tyson seriously off.

With the data from above:

[image]

48 subjects. Agree with “tooth gaps” in small studies. Below ~30 you will see funny patterns.

Cheers,
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. ☼
Science Quotes
Lucas
Regular

Brazil,
2014-02-19 16:13
(edited by Lucas on 2014-02-19 20:22)

@ ElMaestro
Posting: # 12455
Views: 10,756
 

 Qualitative analysis of the ANOVA residuals

Hello!

» [...] it may just look like the teeth of someone who pissed Mike Tyson seriously off.

That is an extraordinary way to describe it! :rotfl:

» Looking for outliers (looking in this regards means looking, and not quantifying their potential presence) may be another aspect where this could come in handy.

I don't know if I'm thinking too "statistically", but that wouldn't be very subjective? One can look and think that there are outliers and other can think that there aren't and both would not be wrong, since there is no criteria.
Anyway, your's is a good way to attend to that requirement from the sponsor, but IMHO a qualitative analysis for the residuals would not add any value to the study, from a regulatory perspective. Am I wrong?
ElMaestro
Hero

Denmark,
2014-02-19 20:40

@ Lucas
Posting: # 12458
Views: 10,725
 

 Qualitative analysis of the ANOVA resid­u­als

Hi Lucas,

» I don't know if I'm thinking too "statistically", but that wouldn't be very subjective? One can look and think that there are outliers and other can think that there aren't and both would not be wrong, since there is no criteria.

You're right, there is some degree of subjectivity in my proposal. I will argue that this is the case for many if not most qualitative assessments, like for instance a CoA for a injectable where you have a spec like "A colorless or slightly yellowish liquid".

» Anyway, your's is a good way to attend to that requirement from the sponsor, but IMHO a qualitative analysis for the residuals would not add any value to the study, from a regulatory perspective. Am I wrong?

You're right, at least per principle. Regulatory decisions are as far as I know not based on residuals or their distributions in any territory. However, regulators may ask for anything they like ("Please submit all chromatograms for QCs and calibrators injected during validation, maintenance records from 2009-2011 for freezer ZX/Y353, and the phone number of the hot brunette from I.P. who attended the last pre-submission meeting" etc), and they may ultimately use that additional information against you as they please.
No idea why a sponsor would ask you to do this, other than if some newly hired numbercruncher wants to impress her/his boss by asking something that sounds complicated. If Helmut's figure above applies to you data then just submit that, tell the sponsor you think this looks reasonably normal (as in Gaussian) without the presence of definitively aberrant values (you will deliberately not use the term outliers).

if (3) 4

Best regards,
ElMaestro

"(...) targeted cancer therapies will benefit fewer than 2 percent of the cancer patients they’re aimed at. That reality is often lost on consumers, who are being fed a steady diet of winning anecdotes about miracle cures." New York Times (ed.), June 9, 2018.
Activity
 Thread view
Bioequivalence and Bioavailability Forum |  Admin contact
19,016 posts in 4,056 threads, 1,297 registered users;
online 19 (2 registered, 17 guests [including 13 identified bots]).

The only people who see the whole picture are the ones
who step out of the frame.    Salman Rushdie

The BIOEQUIVALENCE / BIOAVAILABILITY FORUM is hosted by
BEBAC Ing. Helmut Schütz
HTML5 RSS Feed