jag009
★★★

NJ,
2013-07-24 19:30
(3900 d 13:51 ago)

Posting: # 11051
Views: 19,276
 

 SAS vs Winnonlin [Software]

Hi all,

I have a question about BE study results... Since both SAS and WinNonlin results are submittable for FDA (EMA?), what if my results are borderline passing the 90% CI with SAS and borderline failing the 90% CI with WinNonlin?

Just an example: SAS gives 79.85% - 99.93%, while WinNonlin gives 80.15% - 100.52%.

Thanks
John
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2013-07-24 19:55
(3900 d 13:26 ago)

@ jag009
Posting: # 11052
Views: 16,665
 

 SAS vs Winnonlin

Hi John,

❝ Since both SAS and WinNonlin results are submittable for FDA (EMA?), …


EMA: [image]

❝ … what if my results are borderline passing the 90% CI with SAS and borderline failing the 90% CI with WinNonlin?

❝ Just an example: SAS gives 79.85% - 99.93%, while WinNonlin gives 80.15% - 100.52%.


Some ideas:
  • Conventional 2×2 cross-over: Since Phoenix/WinNonlin only speaks REML, you have to exclude incomplete data. Unlike in SAS GLM (which ignores such cases) you have to filter for complete data in Phoenix first. Otherwise you are in mixed-effects model limbo. For the EMA you have to treat all effects fixed. In Phoenix that means: Delete subject(sequence) in the variance structure / random1 tab and add it to the fixed effects.
  • Higher-order Xover (say two references). Same as above, but for the EMA you have to exclude the “uninteresting” treatment, while for the FDA I guess (!) you keep all in the model and report only the relevant pair. That’s independent from the software you use.
  • Replicate designs / scaling: Cave – FDA and EMA have different models.
With the correct coding I never saw different results from SAS and PHX. Is this a hypothetical example or from the ‘real world’? On the other hand different results in higher-order Xovers are possible (again independent from software). In your example substitute SAS by FDA and WinNonlin by EMA. The former comes from the full model with a pooled variance of all formulations, whereas the latter pools only variances of test and the European reference. If the US RLD has a lower variance than the European reference, in the EMA’s model you will get a wider CI because the ‘dampening effect’ of the US RLD is not applicable.

Dif-tor heh smusma 🖖🏼 Довге життя Україна! [image]
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
jag009
★★★

NJ,
2013-07-25 00:56
(3900 d 08:25 ago)

@ Helmut
Posting: # 11055
Views: 16,520
 

 SAS vs Winnonlin

Hi Helmut,

❝ Some ideas:

Conventional 2×2 cross-over: Since Phoenix/WinNonlin only speaks REML, you have to exclude incomplete data. Unlike in SAS GLM (which ignores such cases) you have to filter for complete data in Phoenix first. Otherwise you are in mixed-effects model limbo. For the EMA you have to treat all effects fixed. In Phoenix that means: Delete subject(sequence) in the variance structure / random1 tab and add it to the fixed effects.

Higher-order Xover (say two references). Same as above, but for the EMA you have to exclude the “uninteresting” treatment, while for the FDA I guess (!) you keep all in the model and report only the relevant pair. That’s independent from the software you use.


It is a 3-way 2 test vs reference study (tada! You guessed it! We have common/pool variance!)
Well the data I have is not complete, one subject is missing period 3 (withdrew) but I elected to keep his periods 1 and 2 data since period 2 was the reference arm. I used GLM in SAS and then for my own amusement I ran the data in Phoenix...

John
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2013-07-25 01:53
(3900 d 07:29 ago)

@ jag009
Posting: # 11058
Views: 16,299
 

 SAS vs Winnonlin

Hi John,

❝ It is a 3-way 2 test vs reference study (tada! You guessed it! We have common/pool variance!)

❝ Well the data I have is not complete, one subject is missing period 3 (withdrew) but I elected to keep his periods 1 and 2 data since period 2 was the reference arm. I used GLM in SAS and then for my own amusement I ran the data in Phoenix...


If you are walking the fun road already: You can feed the incomplete data to PROC MIXED and the other way ’round the complete to PHX. You should see again different results, but in reversed order.

Dif-tor heh smusma 🖖🏼 Довге життя Україна! [image]
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
jag009
★★★

NJ,
2013-07-25 02:56
(3900 d 06:25 ago)

@ Helmut
Posting: # 11060
Views: 16,366
 

 SAS vs Winnonlin

Hi helmut,

❝ If you are walking the fun road already: You can feed the incomplete data to PROC MIXED and the other way ’round the complete to PHX. You should see again different results, but in reversed order.


Yup, did that all. I love chaos and to confuse my direct reports...

Here is question though but I think I might have asked you (and others) before. With a three way x-over study, what if the 1st formulation fails on 90% CI due to the pool variance being inflated by the 2nd formulation? Can I file something to the agency to have them to re-consider the fact that the 1st formulation actually passes BE if the study was conducted as a two way? How? By collapsing the data into a 2-way study while preserving the order of the T and R treatments are per sequence (i.e., ABC → CB, BAC → AC, CBA → CA etc etc) and run stats? Or remove the 2nd formulation data completely and run stats? So that the width of the 90% CI is attributed to intrasubject CV from both 1st formulation and reference.

Just curious that all. I know I know its a bad thing to do. But I am not altering any data or randomness of the treatments (Am I?).

Thanks
John
d_labes
★★★

Berlin, Germany,
2013-07-25 11:13
(3899 d 22:08 ago)

@ jag009
Posting: # 11061
Views: 16,261
 

 EMA crippled approach

Dear John,

let me snap in.

❝ ... How? By collapsing the data into a 2-way study while preserving the order of the T and R treatments are per sequence (i.e., ABC → CB, BAC → AC, CBA → CA etc etc) and run stats? Or remove the 2nd formulation data completely and run stats? So that the width of the 90% CI is attributed to intrasubject CV from both 1st formulation and reference.


If you aimed for an EMA submission you have to do some sort of that.
To cite the EMA guideline again (we had this already occasionally discussed here, also by yourself!):
"In studies with more than two treatment arms (e.g. a three period study including two references, one from EU and another from USA, or a four period study including test and reference in fed and fasted states), the analysis for each comparison should be conducted excluding the data from the treatments that are not relevant for the comparison in question."

Practically this means to me: Don't change anything in the dataset. Keep periods and sequences as they are. Only forgot 1/3 of your data per comparison under consideration.

I'm not a friend of this crippled approach :no:.
But ... On the other hand the full analysis is crucially depending on the assumption of a common error variance for the three treatments. What happens and what are the implications if this assumption is violated is not really clear to me.

Is the case of heteroscedastic variances in BE studies an issue? Don't ask me :smoke:. See a quote of Steven Senn here.

Regards,

Detlew
jag009
★★★

NJ,
2013-07-25 17:38
(3899 d 15:44 ago)

@ d_labes
Posting: # 11062
Views: 16,258
 

 Hypothesis

Hi Detlew,

❝ If you aimed for an EMA submission you have to do some sort of that.

❝ To cite the EMA guideline again (we had this already occasionally discussed here, also by yourself!):

"In studies with more than two treatment arms (e.g. a three period study including two references, one from EU and another from USA, or a four period study including test and reference in fed and fasted states), the analysis for each comparison should be conducted excluding the data from the treatments that are not relevant for the comparison in question."


Sorry I was going to mention that in my post but I got carried away :-)

But you get my point correct? There is no alteration of the study data or the method of analysis. I can present the study data into two sets of results by removing each of the test formulation data from the dataset (yes keeping the period and sequence as a 3-way study) and run stats on each. The point is to show that one formulation actually fails because of the pool variance effect attributed to the high CV from the other formulation and removing the "bad" formulation data results in passing BE.

Of course I have multiple parameters (Cmax and AUCs) to deal with... What if one parameter passes and one (or two) fails after doing the split data analysis...

We actually spoke with FDA about the pool variance effect (at a conference) and they didn't seem to care. Our question was "Would you raise a question about the results of a 3-way '2 formulation vs 1 reference' study because of the pool variance effect?". I wasn't there but if I was then I would have asked "What if the study fails marginally due to pool variance? Can I re-evaluate the data by removing each formulation and run stats to show that one formulation actually passes BE?"

John
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2013-08-10 15:52
(3883 d 17:29 ago)

@ jag009
Posting: # 11260
Views: 16,454
 

 Two at a Time? Or All at Once?

Hi John,

❝ We actually spoke with FDA about the pool variance effect (at a conference) and they didn't seem to care. […]



I found the abstract of a 15 minutes presentation from the 2004 ENAR Spring Meeting of the IBS. Since it is so short below in all its splendor:

TWO AT A TIME? OR ALL AT ONCE?
Donald J. Schuirmann, U.S. Food and Drug Administration

Suppose we have a bioequivalence study with three treatments – A, B, and C – and the objective of the study is to make pairwise comparisons among the treat­ments. Suppose further that treatment C is different in kind from A and B, so that the assumption of homogeneous variance among the three treatments is ques­tionable. One way to do the analyses, under normality assumptions, is Two at a Time – e.g., to test hypotheses about A and B, use only the data from A and B. Another way is All at Once – include the data from all three treatments in a single analysis, making pairwise comparisons within this analysis. If the assump­tion of homogeneous variance is correct, the All at Once approach will provide more d.f. for estimating the common variance, resulting in increased power. If the variance of C differs from that of A and B, the All at Once approach may have reduced power or an inflated type I error rate, depending on the direction of the difference in variances. I will attempt to quantify the difference between the two approaches for both the comparison of A to B and the compa­rison of A or B to C. Both parallel and crossover designs will be considered.

e-mail: [email protected]


Dif-tor heh smusma 🖖🏼 Довге життя Україна! [image]
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2013-07-27 20:30
(3897 d 12:51 ago)

@ d_labes
Posting: # 11076
Views: 16,100
 

 Heteroscedasticity

Dear Detlew,

❝ I'm not a friend of this crippled approach :no:.


Me not either. On the other hand, remember your own simulations?

❝ On the other hand the full analysis is crucially depending on the assumption of a common error variance for the three treatments. What happens and what are the implications if this assumption is violated is not really clear to me.


❝ Is the case of heteroscedastic variances in BE studies an issue? Don't ask me :smoke:. See a quote of Steven Senn here.


Stephen and Andy were taking about study planning and IMHO, they erred. We all know that f.i. generic formulations of PPIs show lower CVs since many references are lousy pro­ducts (mainly problems with occasionally failing gastro­resistant coating). Should we not perform such studies because we expect hetero­scedasticity? Doesn’t make sense to me. Although the author of famous PowerTOST is not entirely happy with the underlying maths, the outcome (assuming different CVWs in replicate designs) matches common sense: A reward in terms of power / sample size if CVWT < CVWR and a penalty if CVWT > CVWR.

Another story are parallel designs (:waving: ElMaestro). Here we deal with the pooled total variance, which will be influenced by different formulation CVs as well, but to a much lesser extent than in Xovers. Differences between groups end up straight in the treatment com­parison. Therefore, it is of paramount importance to standardize studies in a parallel design. Though matched pairs are sometimes used in phase III I don’t know whether such an approach was ever seriously* tried in BE. It would need some a priori knowledge of influential covariates on PK. BMI, body-weight and/or -fat, sex, and renal clearance should perform better than eye’s color. [image]


  • Well, I tried in the last century, but the sample size was too small for meaningful matching. I ran out of degrees of freedom. Was just some kind of post hoc hobby project.

Dif-tor heh smusma 🖖🏼 Довге життя Україна! [image]
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
jag009
★★★

NJ,
2013-07-27 07:59
(3898 d 01:22 ago)

@ Helmut
Posting: # 11066
Views: 16,175
 

 Update!

Hi Helmut and others,

❝ With the correct coding I never saw different results from SAS and PHX. Is this a hypothetical example or from the ‘real world’? On the other hand different results in higher-order Xovers are possible (again independent from software)...


Here is a strange one I just experience. 100% final plasma data from a 3-way (2 formulation vs reference) study. Both myself and the CRO used SAS Proc GLM to compute A vs C, B vs C (alpha=0.05 yes). Results? My lower 90% CI for formulation 1 was 80.24, CRO result showed 79.65 at the lower limit of the 90% CI. I checked my individual parameter values with theirs and they all matched! Okay their report only show 2 decimal places and I use 4 decimals. The values are in tenths (? I mean numbers like 20.50, etc, not into hundredths). The difference in arithmetic means are like .5 for some reason.

Geometric (least square) means are also off
Mines are 31.82 (T), 35.50 (R), Intra CV = 24.44
Theirs are 31.8 (T), 35.55 (R), Intrasub CV = 25.8

??? My suspicion is the number of decimal places they used. The plasma-time data is 3 decimal places. No time deviation (0-24 hrs sampling, all in house)

John

P.S. I tried running WinNonlin and the output confirmed my numbers as well.
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2013-07-27 16:53
(3897 d 16:28 ago)

@ jag009
Posting: # 11075
Views: 16,179
 

 Rounding limbo?

Hi John,

❝ […] My lower 90% CI for formulation 1 was 80.24, CRO result showed 79.65 at the lower limit of the 90% CI. I checked my individual parameter values with theirs and they all matched! Okay their report only show 2 decimal places and I use 4 decimals.

❝ ??? My suspicion is the number of decimal places they used. The plasma-time data is 3 decimal places.


Reasonable assumption. It might well be that – despite concentrations in the report are given with three decimals – the CRO used results in full precision. That’s not uncommon if a CRO runs both the analytical and statistical part of the study or if analytical results are transfer to another site in electronic form (f**ing Excel or SAS transport format). Ask the CRO to repeat the evaluation with data rounded to three decimal places (as given in the report). If they don’t get your results, something is wrong.

BTW, data transfer should always be done to the same precision as given in the analytical report. Imagine a case where a study passes with data in full precision and fails with results given in the report. Yes, regulators sometimes (especially if the CI is borderline) recalculate studies. Will ring the bell. In Excel there is an option to export to a CSV-file with “Precision as displayed”. That’s the way to go if you want to avoid discrepancies.

❝ P.S. I tried running WinNonlin and the output confirmed my numbers as well.


Pharsight will be happy to read this. ;-)

Dif-tor heh smusma 🖖🏼 Довге життя Україна! [image]
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
jag009
★★★

NJ,
2013-07-28 00:02
(3897 d 09:19 ago)

@ Helmut
Posting: # 11077
Views: 15,972
 

 Rounding limbo?

Hi Helmut,

The bioanalytical work was performed by another CRO (contract Lab). The datafile I received was the exact copy that the CRO (one who ran the stats) received. I even spoke with the contract lab people and they confirmed that they reported only 3 decimal places and it was an excel file.

The other strange part is that my formulation B outcome was exactly the same as what they have (down to the 2nd decimal place for ratios, 90%CI and Least square means).

My broadsword is ready for Monday's telecon. :angry:

John
mittyri
★★  

Russia,
2014-01-10 09:05
(3730 d 23:16 ago)

@ Helmut
Posting: # 12153
Views: 15,237
 

 Winnonlin: exclude volunteers

Hi Helmut and All,

Sorry for out-of-date question, but I need to clarify one issue.

❝ Some ideas: Conventional 2×2 cross-over: Since Phoenix/WinNonlin only speaks REML, you have to exclude incomplete data. Unlike in SAS GLM (which ignores such cases) you have to filter for complete data in Phoenix first. Otherwise you are in mixed-effects model limbo.


In the study we have 2 volunteers who were discontinued the study after 1st period (both from same sequence)
Do I understand correct, that I shouldn't put the data from these subjects to WNL?
I tried both options (with and without them) and recieved unequal CIs

Kind regards,
Mittyri
Helmut
★★★
avatar
Homepage
Vienna, Austria,
2014-01-10 14:41
(3730 d 17:40 ago)

@ mittyri
Posting: # 12155
Views: 15,623
 

 Winnonlin: exclude incomplete data!

Hi mittyri,

❝ ❝ […] Since Phoenix/WinNonlin only speaks REML, you have to exclude incomplete data. […] you have to filter for complete data in Phoenix first. Otherwise you are in mixed-effects model limbo.


❝ In the study we have 2 volunteers who were discontinued the study after 1st period (both from same sequence)

❝ Do I understand correct, that I shouldn't put the data from these subjects to WNL?


Yes. See also the EMA’s GL:

[…] subjects in a crossover trial who do not provide evaluable data for both of the test and reference products […] should not be included.


❝ I tried both options (with and without them) and recieved unequal CIs


Correct. PHX/WNL in its default setting (incomplete data included) will recover information of the between-subject variance (REML estimate of a mixed-effects model). In SAS-speak that’s Proc MIXED instead of Proc GLM. In other words, the variance will be differently split into their between-, within-, and residual components. Therefore, the CI (and CVintra) will be affected as well – though in my experience the difference generally is quite small.
See also an example in SAS 9.2 and WinNonlin 5.2.1.

Dif-tor heh smusma 🖖🏼 Довге життя Україна! [image]
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes
ElMaestro
★★★

Denmark,
2013-07-24 20:06
(3900 d 13:15 ago)

@ jag009
Posting: # 11053
Views: 16,532
 

 SAS vs Winnonlin

Hi John,

apart from what Hötzi just said, I think the issue is of no practical importance since you will submit the data analysed in accordance with the protocol and SAP, which in turn specify the software and mode of analysis :lol2:

Pass or fail!
ElMaestro
jag009
★★★

NJ,
2013-07-25 00:59
(3900 d 08:22 ago)

@ ElMaestro
Posting: # 11056
Views: 16,278
 

 SAS vs Winnonlin

Hi ElMaestro,

❝ ...in accordance with the protocol and SAP, which in turn specify the software and mode of analysis :lol2:


I see! Thanks!

PS. Oh get this. There was one CRO who used both WinNonlin and SAS for PK and stats. WinNonlin to compute the PK parameters, SAS to do the stats.

John
UA Flag
Activity
 Admin contact
22,957 posts in 4,819 threads, 1,636 registered users;
86 visitors (0 registered, 86 guests [including 5 identified bots]).
Forum time: 08:22 CET (Europe/Vienna)

With four parameters I can fit an elephant,
and with five I can make him wiggle his trunk.    John von Neumann

The Bioequivalence and Bioavailability Forum is hosted by
BEBAC Ing. Helmut Schütz
HTML5