Log in
Register|
mathews ● 2007-10-11 09:58 (6828 d 23:12 ago) Posting: # 1172 Views: 11,445 |
|
|
Dear All, How we can find the outliers in QC samples?  which test is best for finding the outliers? grubbs test or Dixons test?...why?... Is there any other outlier tests? usually in Bioanalytical application there are 6 QC in each set. some literature says Grubbs test is not applicable if the sample size is less than or equal to 6. but in some papers they mentioned grubbs test is applicable from 3 to 100 sample?...which one is true?... before applying Grubbs test, is it necessary to test whether the samples are normally distributed? thanks and regards mathews  |
|
Helmut ★★★   Vienna, Austria, 2007-10-11 17:31 (6828 d 15:39 ago) @ mathews Posting: # 1173 Views: 10,301 |
|
|
Dear Mathews! ❝ How we can find the outliers in QC samples? You may apply any test only within a concentration level (i.e., run three tests on low/intermediate/high QCs). Please see this thread. According to the FDA (and IMHO to all other regulatory agencies) outlier-tests on QCs are not acceptable. ❝ which test is best for finding the outliers? grubbs test or Dixons test? Dixon Since the test is based on the range of measurements (only the suspected outlier, the minimum and maximum in the series are needed), the more precise your method is, the higher is the likelihood of detecting an outlier. Although tabulated for n≥3, generally 5-8 values are recommended. Grubbs All values in the series are used; although tabulated for n≥3, generally n≥6-8 values are recommended. There's one weakness in Grubbs’ test: if there are two extreme values within a series, they cancel out and no outlier(s) are detected. In such a case the ‘paired Grubbs test’ is recommended. ❝ Is there any other outlier tests? Henning test Let’s denote the suspected outlier by xo, the mean by xm, and the standard deviation by s. Q = |xo - xm| xo is considered an outlier if 3×s ≤ Q. Robust method based on the “outer fence” Q1: first Quartile (25% Percentile) Q3: third Quartile (75% Percentile) IQR = Q3 - Q1: Interquartile Range xo is considered an outlier if xo<Q1 - 3×IQR or xo>Q3 + 3×IQR. Other methods are the MAD (Median Absolute Deviation), Nalimov’s test, Hampel’s test,… For an overview see this article by Shaun Burke (for links to other articles in the series see this post). ❝ usually in Bioanalytical application there are 6 QC in each set. Yes, but at three levels… ❝ some literature says grubbs test is not applicable if the sample size is less than or equal to 6. but in some papers they mentioned grubbs test is applicable from 3 to 100 sample? Both tests are tabulated for n≥3. For recommendations see above. ❝ which one is true?... Concerning ‘truth in science’, see K.R. Popper’s quotation.  Whenever a theory appears to you as the only possible one, ❝ before applying grubbs test, is it necessary to test whether the samples are normally distributed?...
— Dif-tor heh smusma 🖖🏼 Довге життя Україна! ![[image]](https://static.bebac.at/pics/Blue_and_yellow_ribbon_UA.png) Helmut Schütz ![[image]](https://static.bebac.at/img/CC by.png) The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes |
|
yuvrajkatkar ★ Pune, Maharashtra (India), 2010-01-19 09:49 (5997 d 22:21 ago) @ Helmut Posting: # 4615 Views: 9,403 |
|
|
Dear Sir, You have mentioned in this thread, ❝ According to the FDA (and IMHO to all other regulatory agencies) outlier-tests on QCs are not acceptable. It means there is no need for formal 'Outlier Test' for QC's. In which FDA guideline it is mentioned? Edit: Linked to referred thread. Original quote restored. [Helmut] — Best Regards, Yuvraj |
|
H_Rotter ★ Germany, 2010-01-19 13:55 (5997 d 18:14 ago) @ yuvrajkatkar Posting: # 4616 Views: 9,372 |
|
|
Dear yuvrajkatkar! ❝ It means there is no need for formal 'Outlier Test' for QC's. ❝ In which FDA guideline it is mentioned? Do you know Helmut's Guideline collection? For example in the FDA's guidance of 2001. Regards, Hermann |
|
Helmut ★★★ Vienna, Austria, 2010-01-19 18:36 (5997 d 13:34 ago) @ yuvrajkatkar Posting: # 4618 Views: 9,580 |
|
|
Dear Yuvraj! Again: there’s no need for a formal outlier test. Reading the guidance you would have noticed that a QCs sample may be excluded, if:
Consider a simple dataset (n=3; that’s the minimum to do statistics I would say…): QC1 80% (suspected outlier; accuracy <85%)QC2 95% (acceptable: within 100±15%)SD 12.6% (acceptable: <15%)Grubbs-test: 1.060<1.155, p≥0.05Dixon-test: 0.6<0.941, p≥0.05Henning-test: 13.33<37.75, no outlierRobust: no outlierNow what? You may exclude the value according to the guidance, but none of the tests would support that. Not even a “zero” result for QC1 would give you a significant result. BTW, you may exclude values, but you don’t have to. In the example most people wouldn’t do so, because both accuracy and precision are within limits. Forget it. — Dif-tor heh smusma 🖖🏼 Довге життя Україна! Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes |
|
Ohlbe ★★★ France, 2010-01-19 20:00 (5997 d 12:10 ago) @ Helmut Posting: # 4619 Views: 9,467 |
|
|
Dear Helmut, ❝ Reading the guidance you would have noticed that a QCs sample may be excluded, Sorry, but I disagree there. You may exclude a standard sample, but I can't see how you could exclude a QC, unless you have an excellent and documented analytical reason to do it. And when I say documented, I don't mean just writing "sample processing error" in your raw data just because the QC fails, but rather something like no internal standard added, or sample not injected, or chromatographic interference. Your QC result is an experimental result, like it or not, outlier or not. Best regards Ohlbe — Regards Ohlbe |
|
Helmut ★★★ Vienna, Austria, 2010-01-19 20:16 (5997 d 11:54 ago) @ Ohlbe Posting: # 4620 Views: 9,353 |
|
|
Dear Ohlbe! ❝ Sorry, but I disagree there. You may exclude a standard sample, but I can't see how you could exclude a QC, unless you have an excellent and documented analytical reason to do it. And when I say documented, I don't mean just writing "sample processing error" in your raw data just because the QC fails, but rather something like no internal standard added, or sample not injected, or chromatographic interference. Your QC result is an experimental result, like it or not, outlier or not. Disagree. FDA (2001, pages 13 and 16):
That’s exactly my example: one value below –15%, but overall bias and precision within ±15%. Of course the value has to be reported and included in the calculation of acc./prec. I don’t get the rationale why you would allow for exclusion of a calibrator (if not changing the model, according to FDA), but not a QC (both spiked matrix, etc.) BTW, even the term “outlier” with n=2–3 is crazy. — Dif-tor heh smusma 🖖🏼 Довге життя Україна! Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes |
|
Ohlbe ★★★ France, 2010-01-19 22:00 (5997 d 10:10 ago) @ Helmut Posting: # 4621 Views: 9,300 |
|
|
Dear Helmut, ❝ Of course the value has to be reported and included in the calculation of acc./prec. Then we agree: you're not excluding it... Of course I agree that you can have up to 1/3 of your QCs failing. But that's not what I call exclusion. The failing results are there, and should be reported and included in the calculations, whether considered statistically to be outliers or not. Best regards Ohlbe — Regards Ohlbe |
|
Helmut ★★★ Vienna, Austria, 2010-01-19 22:49 (5997 d 09:20 ago) @ Ohlbe Posting: # 4622 Views: 9,382 |
|
|
Dear Ohlbe! ❝ ❝ Of course the value has to be reported and included in the calculation of acc./prec. ❝ ❝ Then we agree: you're not excluding it... ❝ ❝ Of course I agree that you can have up to 1/3 of your QCs failing. OK. ❝ But that's not what I call exclusion. The failing results are there, and should be reported and included in the calculations, whether considered statistically to be outliers or not. Again, an outlier test with n=2–3 is futile. IMHO such a test shouldn’t be used in any case. To be honest I don’t know of any analyst even trying one. Well, and results should be reported. But: In this post you wrote: ❝ You may exclude a standard sample, but I can't see how you could exclude a QC, unless you have an excellent and documented analytical reason to do it. ❝ […] Your QC result is an experimental result […] Again, what’s your rationale of “excluding a standard sample” – calibrators and QCs are prepared in exactly the same way? Both are spiked samples of known concentrations. If you ask for an “excellent and documented analytical reason” to exclude a QC, why do you not require the same for standards? Or do you? To paraphrase your statement: Your calibrator is an experimental result […] — Dif-tor heh smusma 🖖🏼 Довге життя Україна! Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes |
|
Ohlbe ★★★ France, 2010-01-20 20:01 (5996 d 12:09 ago) @ Helmut Posting: # 4628 Views: 9,286 |
|
|
Dear Helmut, ❝ Again, what’s your rationale of “excluding a standard sample” – calibrators and QCs are prepared in exactly the same way? Both are spiked samples of known concentrations. If you ask for an “excellent and documented analytical reason” to exclude a QC, why do you not require the same for standards? Or do you? To paraphrase your statement: Your calibrator is an experimental result […] True. But calibrators and QCs have a different aim in your run:
Ohlbe — Regards Ohlbe |
|
Helmut ★★★ Vienna, Austria, 2010-01-20 20:26 (5996 d 11:44 ago) @ Ohlbe Posting: # 4629 Views: 9,298 |
|
|
Dear Ohlbe! ❝ ❝ Again, what’s your rationale of “excluding a standard sample” – calibrators and QCs are prepared in exactly the same way? Both are spiked samples of known concentrations. If you ask for an “excellent and documented analytical reason” to exclude a QC, why do you not require the same for standards? Or do you? To paraphrase your statement: ❝ ❝ Your calibrator is an experimental result […] ❝ ❝ True. But calibrators and QCs have a different aim in your run: ❝ - calibrators are used to calculate all concentrations in your run (subject samples and QCs). ACK. ❝ A problem with a single calibrator will directly and negatively affect all results in the run. Sidenote: As may a single QC.  ❝ And what you're trying to measure is not the back-calculated concentration of your calibrators (which you fundamentally don't care about)… I’m interested in the back-calculated concentrations as well. OK, to be more precise, to check the residuals (haha, is the applied weighting scheme of the curve appropriate as established in validation). I have to back-calculate concentrations first, in order to be able to exclude a point. Again, would you allow that without justification (in my limited experience FDA does), or would you like to see the same justifications you required for QCs? ❝ … but the concentration of your subject samples. Right, that’s the primary objective. ❝ - QCs are there to monitor the precision and accuracy of your method, as applied to your subject samples. Treating them differently from your subject samples would therefore introduce a bias in that estimation. I didn’t ask for that. I got the impression that you would treat calibrators and QCs differently. ❝ […] many Agencies unfortunately hate to hear about "PK repeats". Yes, you know my opinion about that: It’s not only bad science, but in contradiction to many authorities as well (FDA, HPB/TGD, ANVISA,…). So much about harmonization. ❝ So unless you have a clear analytical reason, you will accept all subject samples in a validated analytical run. Sure. ❝ The same goes with QC samples… What do you mean by this? Sounds like an oxymoron to me. P.S.: Interesting thread. — Dif-tor heh smusma 🖖🏼 Довге життя Україна! Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes |
|
Ohlbe ★★★ France, 2010-01-20 20:52 (5996 d 11:18 ago) @ Helmut Posting: # 4630 Views: 9,273 |
|
|
Dear Helmut, ❝ Sidenote: As may a single QC. Yes, sure  ❝ I’m interested in the back-calculated concentrations as well. OK, to be more precise, to check the residuals (haha, is the applied weighting scheme of the curve appropriate as established in validation). I have to back-calculate concentrations first, in order to be able to exclude a point. OK, agreed, of course. But that's not the primary goal of the study. ❝ Again, would you allow that without justification (in my limited experience FDA does), or would you like to see the same justifications you required for QCs? Yes, I would accept the exclusion of a calibration sample without any "assignable cause", as the FDA would put it. ❝ I got the impression that you would treat calibrators and QCs differently. I do. I accept calibrators to be excluded from the calibration curve without an "assignable cause", if they are more than 15 % from nominal (20 % at the LLOQ). The back-calculated concentration should be reported in the analytical report even if excluded (clearly identified as excluded), but I don't mind if it is not included in the calculation of the precision and accuracy of the results at each level of concentration (actually I don't mind if the mean and CV of the back-calculated concentration of the calibrators at each level of concentration are not calculated at all - that's not really relevant). But I do consider that all QCs should be included in the calculations, unless you had a true analytical problem (in which case I would not expect the result to be reported at all). ❝ ❝ The same goes with QC samples… ❝ ❝ What do you mean by this? Sounds like an oxymoron to me. Once again: I treat QCs the same way as unknown samples. Report them all unless you had a clear analytical problem, include them all into calculations, no exclusion from the report, or they won't be representative any more. ❝ P.S.: Interesting thread. Yes Best regards Ohlbe — Regards Ohlbe |
|
H_Rotter ★ Germany, 2010-01-20 17:21 (5996 d 14:49 ago) @ Ohlbe Posting: # 4624 Views: 9,217 |
|
|
Dear Ohlbe, I am not an expert in analytics, so I'm confused. ❝ Of course I agree that you can have up to 1/3 of your QCs failing. But that's not what I call exclusion. The failing results are there, and should be reported and included in the calculations,... I've read FDA's guidance again (let's talk not about values at LLOQ): in method validation mean value ≤±15% of nominal, precision given as SD/mean ≤±15% CV (≥5 replicates). Reading the section on routine analysis over, I couldn't find anything about precison of QCs (determined from at least duplicates) - only bias. OK, the second paragraph on page 14 states: 'Once the analytical method has been validated for routine use, its accuracy and precision should be monitored regularly to ensure that the method continues to perform satisfactorily.' At the bottom of page 15: 'Acceptance criteria for accuracy and precision as outlined in section IV.F, "Specific Recommendation for Method Validation,” should be provided for both the intra-day and intra-run experiment.' But on the other hand top of page 16 gives the 2/3 statement - only bias, nothing about precision. I always thought that both bias and precison should be evaluated; now I'm not sure anymore and left confused. Let's consider a usual method - ±15% were established in validation (bias and precision) and continue Helmut's example: QC1 bias -20% but mean (93.3%) and precision (13.5%) within limits. Now let's decrease QC1 further down (keeping results of QC2 and QC3): the limit of 15% CV is reached if QC1 = 77.5%, the limit of -15% bias of the mean is reached if QC1 = 55.0%. If limits are ±15% anyhow, why all the fuzz with the 2/3 statement? Or does it mean single QCs may be outside ±15%, if the mean and the CV is ±15%? Regards, Hermann |
|
Ohlbe ★★★ France, 2010-01-20 19:49 (5996 d 12:21 ago) @ H_Rotter Posting: # 4627 Views: 9,262 |
|
|
Dear Hermann, In the pre-study validation the precision and accuracy of the method are monitored, both within each validation run and between run. Actually what is measured is the overall precision and the overall accuracy, not strictly speaking the between-run component of the variability. The terminology used is a bit confusing and I guess it would hurt pure statisticians, but that's what everybody's using. The FDA guideline gives criteria for both precision and accuracy (15 % for chromatographic methods). During the study phase the acceptance criteria defined in the FDA guideline are only given for individual QC results, and there are no acceptance criteria for the mean and CV either intra-run (which would make little sense with possibly only two results at each level of concentration) or globally. Each individual QC result is affected simultaneously by the precision and the accuracy of your method (some call it the total error), and that's how you monitor them. There is a vague statistical basis behind the 2/3 figure. Let's suppose that your QC results have a normal distribution. About 32 % of your QC results will deviate from the mean from 1 SD and 5 % will deviate from the mean by 2 SD. To make things easier, let's suppose that your observed mean is 100 % accurate (identical to the nominal concentration). You allow a CV of 15 %: SD will be 15 % of the mean. So with a CV of 15 % you will get 32 % of your QCs (rounded to 1/3) deviating from the nominal concentration from more than 15 %... And this has been used to define acceptance criteria for 6 QC results at 3 levels of concentration, each with its own distribution... Things are a bit different in the draft EMA guideline, where global acceptance criteria have been defined (15 % for the overall precision and accuracy). Best regards Ohlbe — Regards Ohlbe |
|
moblak ☆ 2010-01-25 11:04 (5991 d 21:06 ago) @ Ohlbe Posting: # 4644 Views: 9,182 |
|
|
Dear all, A very interesting topic! Could someone please comment the extreme example explained below: During the study, we normally have 2 QC replicates at three diff. QC levels for each run. Our QC acceptance criteria for each run: at least 4/6 of all QCs and 1/2 QCs at each level must be within ±15% of the nominal concentration (%Bias). In addition, we have a during-study acceptance criteria for all QCs at each level: %CV<15% and %Nominal 100±15%. Now the »extreme« example; nearly »ideal« method… a study of 50 subjects….50 runs…. 100 QCs at each level 49 runs…. all QCs at each level have %Nominal of 100% 1 run… 5 QCs 100% Nominal and one QC at level 1 300% Nominal (run is accepted) During the investigation of this run you don't find any analytical reason to exclude »300%QC« (chromatography OK, IS response consistent,….) … you suspect a possible sample switch or problem (contamination?) during extraction…but you can't prove that. All runs within study are accepted according to the 4/6 and 1/2 rule; however, if we calculate %CV and %Bias, we get the following results:
QC level 1 %CV=20(19.6); %Nominal=102 (not acceptable)According to the FDA guidelines (“Summary information on intra- and inter-assay values of QC samples and data on intra- and inter-assay accuracy and precision from calibration curves and QC samples used for accepting the analytical run.“), this nearly “ideal” method would be unsuitable due to only one out of 300 QCs failing really badly. But is this method really not OK? In general I agree with “no outliers policy”, but on the other hand, FDA “allows” some sort of the outliers: “Reported method validation data and the determination of accuracy and precision should include all outliers; however, calculations of accuracy and precision excluding values that are statistically determined as outliers can also be reported.” If the “300%QC” would be rejected as an outlier, the method would become ideal. Any ideas? Regards Marko p.s. I tried to write it shorter, but I just couldn’t manage it. |
|
Ohlbe ★★★ France, 2010-01-26 00:19 (5991 d 07:51 ago) @ moblak Posting: # 4646 Views: 9,135 |
|
|
Dear Marko, Thanks for the nice extreme example... ❝ Now the »extreme« example; nearly »ideal« method…a study of 50 subjects….50 runs….100 QCs at each level 49 runs….all QCs at each level have %Nominal of 100% ❝ ❝ 1 run… 5 QCs 100% Nominal and one QC at level 1 300% Nominal (run is accepted) Shit happens  ...❝ In general I agree with “no outliers policy”, but on the other hand, FDA “allows” some sort of the outliers: “Reported method validation data and the determination of accuracy and precision should include all outliers; however, calculations of accuracy and precision excluding values that are statistically determined as outliers can also be reported.” Agreed. Ending in a case-by-case decision, hoping that the assessor also uses his brain and not just the guideline. ❝ If the “300%QC” would be rejected as an outlier, the method would become ideal. Agreed again. But then, the aim of the QCs is precisely to find whether the method is really ideal or not. Thus the requirement to primarily show the data with the "outlier". ❝ During the investigation of this run you don't find any analytical reason to exclude »300%QC« (chromatography OK, IS response consistent,….) … you suspect a possible sample switch or problem (contamination?) during extraction…but you can't prove that. That's often the case. But that's precisely what worries me: how many other samples could be affected ? Not only QCs, but also subject samples ? ❝ p.s. I tried to write it shorter, but I just couldn’t manage it. No way to make it shorter... Regards Ohlbe — Regards Ohlbe |
Ing. Helmut Schütz