Eva
☆    

Germany,
2008-05-06 12:04

Posting: # 1817
Views: 20,137
 

 SAS vs. WinNonlin: different sequence effect results [Software]

Dear all,

I have recently analysed a simple crossover bioequivalence study in SAS and detected a significant sequence effect (p=0.0425). To verify my results, I have run the same analysis using the WinNonlin Bioequivalence Wizard (WNL v5.1.1). The results of the analysis were completely identical except for the sequence effect which was not significant (p=0.0634).

The Pharsight support claims that this is due to a slightly different model used by the Bioequivalence Wizard and recommended me to read the online help on linear mixed effects modelling.
I can not follow this argumentation, because in my opinion my model specification is the same. Besides I get identical results for all other effects: both my SAS program and the BE Wizard produce the same SS, MS and DF for sequence effect, period effect, treatment effect and subject within sequence, the MSE is identical in both programs and (consequently) also the confidence limits.
The F statistics and the p-values are the same for period effect, treatment effect and subject within sequence.

I have repeated the comparison with other data with the same result, all values identical, but the F statistic of the sequence effect always lower in WNL compared to SAS.

Has any of you ever come across this discrepancy before?
I wonder if somebody could help me to find out if there is a mistake in the Bioequivalence Wizard calculation or if I am simply too stupid to model my data correctly...

Thanks in advance & kind regards,
Eva
Ohlbe
★★★

France,
2008-05-06 13:56

@ Eva
Posting: # 1821
Views: 16,095
 

 SAS vs. WinNonlin: different sequence effect results

Dear Eva,

I am not a SAS or WinNonlin specialist, but I would have a basic question. Where do you read the SAS F and p values from ? Usually the printouts I see from SAS present the results in two parts. You first have the full ANOVA with sequence, subject(seq), period and treatment (with F and p values), then a specific test for sequence using the MS for subject(seq) as an error term, (again with a F and p value). Only the results of this specific test are supposed to be considered.

Could this be the origin of the difference you found ?

Regards
Ohlbe
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2008-05-06 14:13

@ Eva
Posting: # 1822
Views: 16,060
 

 imbalanced design?

Dear Eva!

One example in SAS…
                  SUM OF
                  SQUARES     MEAN
EFFECT            TYPE I  DF SQUARE  F-VALUE P-VALUE
----------------------------------------------------
TREATMENT           0.956  1   0.956   7.961   0.007
SUBJ. WITHIN SEQ.  17.802 46   0.387   3.224   <.001
PERIOD              0.100  1   0.100   0.835   0.366
SEQUENCE            0.694  1   0.694   1.794   0.187
ERROR               5.521 46   0.120


… and WinNonlin 5.2
Partial Sum of Squares
Hypothesis      DF    SS       MS     F_stat  P_value
-----------------------------------------------------
SEQUENCE         1  0.694182 0.694182 1.79377  0.1870
SEQUENCE*SUBJ.  46 17.8019   0.386997 3.22418  0.0001
TREATMENT        1  0.955579 0.955579 7.9612   0.0070
PERIOD           1  0.100213 0.100213 0.834903 0.3656
Error           46  5.52136  0.12003


Obviously sequence effects are identical (p 0.187). Maybe you had some drop-outs in your study, which resulted in imbalanced sequences. In such a case SAS’ SS Type III and WinNonlin’s Partial Sum of Squares may differ (although they ‘coincide in most situations’). SAS SS III is a strange pot of tea anyhow… ;-)

It would be nice, if you would come up with a simple dataset (12 subjects?), where you find differences. It should be possible to track it down!

Cheers,
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. ☼
Science Quotes
Eva
☆    

Germany,
2008-05-06 15:42

@ Helmut
Posting: # 1824
Views: 15,906
 

 Sample data & results

Dear Ohlbe and Helmut,

thanks for your fast replies!
Here is a sample data set of 8 subjects, maybe this can help to illustrate my problem...
Subject Period  Sequence Treatment       AUC
------------------------------------------------
1       1       TR       Test            3981.76
2       1       TR       Test            5573.62
3       1       TR       Test            5821.52
4       1       TR       Test            8558.99
5       1       RT       Reference       3958.93
6       1       RT       Reference       5145.91
7       1       RT       Reference       689.6
8       1       RT       Reference       3812.02
1       2       TR       Reference       4446.05
2       2       TR       Reference       4847.12
3       2       TR       Reference       4274.72
4       2       TR       Reference       4839.49
5       2       RT       Test            3219.92
6       2       RT       Test            4587.32
7       2       RT       Test            4436.7
8       2       RT       Test            6168.8


My SAS results (PROC GLM) for these data are
EFFECT             DF    TYPE III SS    MEAN SQUARE     F-VALUE     P-VALUE
---------------------------------------------------------------------------
Sequence           1     0.6105         0.6105          2.46        0.1678
Subject(Sequence)  6     1.5739         0.2623          1.06        0.4739
Treatment          1     0.5365         0.5365          2.16        0.1918
Period             1     0.0774         0.0774          0.31        0.5966
Error              6     1.4887         0.2481


and this is what WinNonlin calculated:
EFFECT             DF    TYPE III SS    MEAN SQUARE     F-VALUE     P-VALUE
---------------------------------------------------------------------------
Sequence           1     0.6105         0.6105          2.33        0.1780
Subject(Sequence)  6     1.5739         0.2623          1.06        0.4739
Treatment          1     0.5365         0.5365          2.16        0.1918
Period             1     0.0774         0.0774          0.31        0.5966
Error              6     1.4887         0.2481


Dear Ohlbe,
I think I have picked the right F- and p-values from my output, I am using PROC GLM (and I just checked, the output is identical with PROC ANOVA) and picked the Sequence, Subject(Sequence), Treatment and Period results from the bottom part and the Error results from the top part.

Dear Helmut,
could this be a problem with my WNL version? (I noticed that you used v5.2, whereas I have v5.1.1)
As you can see above, the sums of squares are not the problem, it is only the F- and p-values that differ.
I understood that the F-value is the ratio of the effect mean square and the mean square error. This is correct for the SAS results (sequence: 0.6105/0.2481=2.46), but not for WinNonLin...

Any ideas?
Thanks again, for your help and for sharing your precious time with my "little" problem...

Kind regards,
Eva


Edit: Treatment in the table changed to Test to avoid confusion with the heading. [Helmut]
Ohlbe
★★★

France,
2008-05-06 17:25

@ Eva
Posting: # 1825
Views: 15,914
 

 Error factor

Dear Eva,

F is the ratio of the mean square of your factor, divided by the mean square of what you consider as the error factor. SAS divided the MS of the sequence effect (0.6105) by the MS of the residual error (0.2481). WinNonlin divided the sequence MS by the MS of subject(sequence) (0.2623). This is what SAS does in the second step I mentioned previously.

Helmut, I will let you explain why subject(sequence) should be used as an error term ;-)

Regards
Ohlbe
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2008-05-06 19:33

@ Ohlbe
Posting: # 1826
Views: 16,154
 

 Error tests between/within

Dear Eva & Ohlbe!

» Helmut, I will let you explain why subject(sequence) should be used as an error term ;-)

OK, sequences are a between subject effect and therefore need to be compared to the between subject residual error (or F=0.6105/0.2623=2.327). In other words we compare (the group of) subjects in sequence TR to (the group of) subjects in sequence RT.

If you compare the sequence against the within subject residual error (0.2481) you cer­tainly will get a different – and wrong! – answer. Using the within subject residual actually overestimates the effect of sequence leading to a greater chance of concluding a sequence effect.

Since in my first post I showed identical results for SAS and WinNonlin, I would expect a coding issue in the SAS program you are using.
Results in WinNonlin-versions don't differ. Unfortunately it’s not possible to have different versions running on the same machine – I re-run ‘old’ evaluations everytime I update to a newer version – and did not get differences since v4.0.1 (3.3 and earlier versions used ANOVA instead of linear mixed effects modeling).

Just to be more comfortable about the results – a little roundup of your dataset:
WinNonlin5.2 (2007)
----------------------------------------------------------
Hypothesis       DF  SS        MS        F_stat    P_value
Sequence          1  0.61054   0.61054   2.32748   0.1780
Sequence*Subject  6  1.57391   0.262318  1.05726   0.4739
Treatment         1  0.536468  0.536468  2.1622    0.1918
Period            1  0.077422  0.077422  0.312045  0.5966
Error             6  1.48867   0.248112


Chow & Liu (2001) – coded in Excel
-------------------------------------------------------
Analysis of Variance                     
Source of variation df  SS      MS       F      P-value
Inter-subjects            
  Carry-over         1  0.6105  0.61054  2.327  0.17795
  Residuals          6  1.5739  0.26232  1.057  0.47392
Intra-subjects
  Direct drug        1  0.5365  0.53647  2.162  0.19184
  Period             1  0.0774  0.07742  0.312  0.59664
  Residuals          6  1.4887  0.24811
Total               15  4.2870


EquivTest/PK (2006)
------------------------------------------------
Analysis of variance table:
               df  SS      MS     F      P-Value
Inter-Subjects
  Carry-over    1  0.6105  0.6105 2.3274 0.1779
  Residuals     6  1.5739  0.2623 1.0572 0.4739
Intra-Subjects
  Drug          1  0.5364  0.5364 2.1622 0.1918
  Period        1  0.0774  0.0774 0.3120 0.5966
  Residuals     6  1.4886  0.2481
Total          15  4.2870


Ref [3] (2001) from this post; coded in R2.7.0 (2008)
-------------------------------------------------------
Error: subf
          Df  Sum Sq Mean Sq F value Pr(>F)
seqf       1 0.61054 0.61054  2.3275 0.1780
Residuals  6 1.57391 0.26232

Error: Within
          Df  Sum Sq Mean Sq F value Pr(>F)
treatment  1 0.53647 0.53647  2.1622 0.1918
Between    1 0.07742 0.07742  0.3120 0.5966
Residuals  6 1.48867 0.24811


NCSS (2001) (Cross-Over Analysis)
-----------------------------------------------
           Estimated  Standard  T Value  Prob
Parameter  Effect     Error     (DF=6)   Level
Treatment  0.3662     0.2491    1.4704   0.1918
Period     0.1391     0.2491    0.5586   0.5966
Carryover  0.7814     0.5122    1.5256   0.1780


Kinetica 4.4.1 (2007)
--------------------------------------------------------
SOURCE      D.F  SS        MS        F          p
Period        1  0.077422  0.077422  0.312045  0.5966 NS
Subject(Seq)  6  1.57391   0.262318  1.05726   0.4739 NS
Formulation   1  0.536468  0.536468  2.1622    0.1918 NS
Sequence      1  0.61054   0.61054   2.46074   0.1678 NS
Error         6  1.48867   0.248112
Total        15  4.28701


[image]Besides your SAS code only Kinetica tests against the within-subject residual error; this a well known bug since v1.1 (‼) – and I would expect ThermoScientific never to correct it… :angry:

Cheers,
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. ☼
Science Quotes
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2008-05-07 16:18

@ Helmut
Posting: # 1830
Views: 15,939
 

 Error tests between/within

» … Unfortunatelly it’s not possible to have different versions running on the same machine…

OK, I remembered WinNonlin4.1 to be still installed on an old machine;
again same results for the sequence effect:

WinNonlin4.1 (2003) Partial Tests of Model Effects
Hypothesis  Numer_DF  Denom_DF   F_stat     P_value
---------------------------------------------------
       int         1         6  4253.66      0.0000
  Sequence         1         6     2.32748   0.1780
 Treatment         1         6     2.1622    0.1918
    Period         1         6     0.312045  0.5966

Cheers,
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. ☼
Science Quotes
d_labes
★★★

Berlin, Germany,
2008-05-08 09:16

@ Helmut
Posting: # 1834
Views: 16,007
 

 The power to know

Dear Eva & Ohlbe & Helmut

» Since in my first post I showed identical results for SAS and WinNonlin, I would expect a coding issue in the SAS program you are using.

That is totally right and wrong.
The SAS Procedure GLM has the "standard" of treating all effects as fixed effects.
Thus if you use the code:
Proc GLM data=test;
  class treatment period sequence subject;
  model logAUC=treatment period sequence subject(sequence) /SS3;
Run;

you get Eva's result. This result is correct if all effects are considered fixed. That means regarding the subject effect we are making inference about the subjects which are actually under study.

But the goal of our BE study is to broaden the scope and make inference about some "population" of subjects (healthy volunteers f.i. from which we have randomly sampled our subjects actually under study).
This leads to subject as a random factor (effect) in our model.

The code in Proc GLM for that is:
Proc GLM data=test;
  class treatment period sequence subject;
  model logAUC=treatment period sequence subject(sequence);
  random subject(sequence) /test;
Run;

Proc GLM is resistant to our wishes and first displays the fixed effect analysis again (this is because according to SAS GLM handles random effects in a "post hoc manner", in direct words this feature was invented later and tinkered into GLM "mit heißer Nadel").

But then the sun is shining bright ;-) (emphasis by me):
The GLM Procedure
Tests of Hypotheses for Mixed Model Analysis of Variance

Dependent Variable: logAUC

Source             DF   Type III SS   Mean Square   F Value   Pr > F

Treatment           1      0.536468      0.536468     2.16    0.1918
Period              1      0.077422      0.077422     0.31    0.5966
Subject(Sequence)   6      1.573910      0.262318     1.06    0.4739
Error: MS(Error)    6      1.488671      0.248112

Source             DF   Type III SS   Mean Square   F Value   Pr > F

Sequence            1      0.610540      0.610540     2.33    0.1780
Error               6      1.573910      0.262318
Error: MS(Subject(Sequence))


Hope this helps a little bit in "the power to know".
SAS is an outstanding beast :-D .

Side note: The annoying fixed effects analysis displayed again is not needed and can be suppressed using the "famous" ODS (whatever this is). Final code:
ODS exclude ExpectedMeanSquares ModelANOVA;
Proc GLM data=test;
  class treatment period sequence subject;
  model logAUC=treatment period sequence subject(sequence);
  random subject(sequence) /test;
Run;

Regards,

Detlew
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2008-05-08 15:18

@ d_labes
Posting: # 1837
Views: 15,957
 

 The power to know

Dear DLabes!

» SAS is an outstanding beast…

:clap:

» […] using the "famous" ODS (whatever this is).

Though I don’t have any idea about SAS, Google is my friend – coming up with ODS (Output Delivery System)

Cheers,
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. ☼
Science Quotes
d_labes
★★★

Berlin, Germany,
2008-05-09 07:35

@ Helmut
Posting: # 1840
Views: 15,709
 

 The power to know

Dear Helmut,

» Google is my friend – coming up with ODS = Output Delivery System…

but this is only for THE INITIATED dragon fighter struggling with the beast :-P.

Regards,

Detlew
Nirali
★    

India,
2008-05-09 09:00
(edited by Nirali on 2008-05-10 02:30)

@ d_labes
Posting: # 1842
Views: 15,828
 

 The power to know

Dear Friend,

Yes, WinNonlin & SAS are giving same outputs for all effects of ANOVA all the time, just need to modify the SAS programm according to the requirement.

Below programm will definitely help you :ok:.

PROC GLM DATA=TEST;
CLASS SEQUENCE SUBJECT PERIOD TREATMENT;
MODEL AUC=SEQUENCE SUBJECT(SEQUENCE) TREATMENT PERIOD /SS3;
TEST H=SEQUENCE E=SUBJECT(SEQUENCE);
RUN;
QUIT;


» SAS is an outstanding beast :-D.

Now, no need to struggle with beast :-) just read below book to get better knowledge of SAS PROGRAMS with basic statistical methods, which is published by SAS INSTITUTE ONLY.

Book: "Common Statistical Methods for Clinical Research with SAS examples" (2nd eddition)
Author: Glenn A. Walker

Thanks & Regards,
Nirali
d_labes
★★★

Berlin, Germany,
2008-05-16 06:54
(edited by d_labes on 2008-05-16 07:26)

@ Nirali
Posting: # 1850
Views: 15,592
 

 The power to know

Dear Nirali,

» Now, no need to struggle with beast :-) just read below book to get better knowledge of SAS PROGRAMS ...

But what to do if I am not able to read?
I have done my scientific work all the time over years (approx. 35 up to now) without reading any page, book or paper :lol3:.
Especially for struggling with the beast it was not necessary for me to have a look into any book or even not into the the documentation. I am an INITIATED! :-P
Thus, don't hurt me with a newbies book at the end of my career.

» Below programm will definitely help you :ok:.

In doing what? Your mentioned code is a little bit outstanding and has its origin in SAS versions lacking the RANDOM statement. The version of GLM code above in this thread with the RANDOM statement has the benefit to make clear the nature of effects (fixed and / or random) under analysis.
It opens the door in understanding the evaluation of more complex designs like replicate x-over using PROC MIXED in SAS or other "Mixed model" ANOVA software.

Regards,

Detlew
kevan
☆    

2009-05-25 13:46

@ d_labes
Posting: # 3749
Views: 14,934
 

 The power to know

» This result is correct if all effects are considered fixed.

Hi d_labes,

Possible all effects are considered fixed especially in nowadays population?

Thanks.
ElMaestro
★★★

Belgium?,
2009-05-25 20:25
(edited by ElMaestro on 2009-05-25 20:37)

@ kevan
Posting: # 3753
Views: 15,148
 

 Bogus statement for 2,2,2-BE

Oi, this was an old thread.

» Possible all effects are considered fixed especially in nowadays population?

The standard 2,2,2-BE model can be fit with a linear model or a mixed model (equivalently glm, lm, lme etc in R) and you can obtain the same results.
When one uses the RANDOM statement in connection with PROC GLM, it means.......very little of relevance to the 2,2,2-B situation, as I see it. It can be omitted and we're still happy. A model with additional random effects other than the good old epsilon residuals is a mixed model, and mixed models are not linear models (they are LMS = "Linear Models on Steroids"). PROC GLM fits a linear model (only!) and in the fiting process it treats all effects as fixed regardless of the presence of a RANDOM statement.
The SAS documentation has the power to confuse: " ...the random effects are treated in a post hoc fashion after the complete fixed effect model is fit. This distinction affects other features in the GLM procedure, such as the results of the LSMEANS and ESTIMATE statements. These features assume that all effects are fixed, so that all tests and estimability checks for these statements are based on a fixed effects model, even when you use a RANDOM statement. Standard errors for estimates and LS-means based on the fixed effects model may be significantly smaller than those based on a true random effects model; in fact, some functions that are estimable under a true random effects model may not even be estimable under the fixed effects model. Therefore, you should use the MIXED procedure to compute tests involving these features that take the random effects into account."

Eh...."True random effects model"? OK, so PROC MIXED is for true random effects models, while PROC GLM is for ...what?....false random effect models, perhaps? (-which presumably are fixed because they don't contain truly random effects)? Aha, now things are very clear.
Does anyone know if there are some handy statements for nonlinear mixed effects models which are not truly linear but also not nonlinear but only sort of a little bit possibly nonlinear on a foggy day, and which can be included or omitted without consequences?

EM.
d_labes
★★★

Berlin, Germany,
2009-05-27 06:57

@ ElMaestro
Posting: # 3773
Views: 15,016
 

 Bogus? What?

Dear ElMaestro,

» The standard 2,2,2-BE model can be fit with a linear model or a mixed model (equivalently glm, lm, lme etc in R) and you can obtain the same results.
» When one uses the RANDOM statement in connection with PROC GLM, it means.......very little of relevance to the 2,2,2-B situation, as I see it.
» It can be omitted and we're still happy.

Out of a walnut shell :-D you are right.

As long as you don't talk about a test of sequence effects.
The RANDOM statement produces the correct test automatically.
See above in this thread to notice that this is of relevance for a number of users (also with other software :-P ), which are not so happy as you.

And as long as you don't talk about the educational benefit of specifying which of your effects you consider fixed or random, which opens you the way to mixed models software necessary for the evaluation of more complicated study designs.

I must confess that I never understand this SASophylistic "... post hoc fashion ...".
But what is actually done in Proc GLM is nowadays known as "method of moments" in the mixed model context.

BTW: Could you explain your "Linear model on Steroids" a bit more in detail? I couldn't get the the joke.

Regards,

Detlew
ElMaestro
★★★

Belgium?,
2009-05-28 17:12
(edited by ElMaestro on 2009-05-29 15:38)

@ d_labes
Posting: # 3784
Views: 14,885
 

 Linear model on steroids

Hi dlabes,

» BTW: Could you explain your "Linear model on Steroids" a bit more in detail? I couldn't get the the joke.

In matrix notation, the normal linear model is written y = Xb + e, and the normal linear mixed model is y = Xb + Zg + e,
i.e. the mixed one is just the linear one plus the Zg-term (model matrix for the random effects and the corresponding random effects vector). "On steroids" is a phrase used to indicate that something has been extended/inflated/improved/enlarged/souped-up etc.

Finally, your explanation about the bogus term is very fine - I did not know that it results in an anova table with the correct F/P-value for the sequence effect. But I think it misses my point slightly. To me, manuals/documentation is where I look and read when I am faced with a problem. How do I etc and why. Let's say you are in a situation where you need to analyse a 2-treatment, 4-period BE study. OK, so you need a mixed model. But should you go for PROC MIXED or PROC GLM in combo with the bogus statement? Does the documentation reffed above help chosing here? I think it doesn't; in fact I think it adds more confusion than it helps. One of the old jokes about the Delphi programming language was that you needed to be an expert Delphi programmer in order to understand the Delphi manual. You see a parallel?

Best regards
EM.

PS: I would use MIXED.
d_labes
★★★

Berlin, Germany,
2009-05-27 07:03

@ kevan
Posting: # 3774
Views: 14,837
 

 Fixed nowadays what?

Dear Kevan,

» Possible all effects are considered fixed especially in nowadays population?

Sorry, but I can not get your point? What is the question? :ponder:
What is your "nowadays population"?

Regards,

Detlew
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2008-12-31 15:42

@ Helmut
Posting: # 2995
Views: 15,916
 

 Kinetica 5.0 bug

Dear all!

» [image][…] only Kinetica tests against the within-subject residual error; this a well known bug since v1.1 (‼) – and I would expect ThermoScientific never to correct it… :angry:

Expectation fulfilled. The bug remained uncorrected in the new release (v5.0 of 19 Dec 2008)… :not really:

Cheers,
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. ☼
Science Quotes
Activity
 Admin contact
20,103 posts in 4,243 threads, 1,382 registered users;
online 8 (0 registered, 8 guests [including 6 identified bots]).
Forum time (Europe/Vienna): 01:45 UTC

We must be careful not to confuse data with the abstractions
we use to analyze them.    William James

The Bioequivalence and Bioavailability Forum is hosted by
BEBAC Ing. Helmut Schütz
HTML5