Helmut ★★★ Vienna, Austria, 2020-05-15 14:31 (1670 d 04:38 ago) Posting: # 21440 Views: 7,879 |
|
Dear all, a colleague received a sample size estimation of an CRO performed in PASS 15.0.5 for a fully replicated study (TRRT|RTTR), CV 0.50, T/R 0.95, power 0.80, ABE (unscaled). The result was N=54 (power 0.8053). Since he is a fan of PowerTOST he tried
54 subjects instead of 50 are good for the CRO, bad for the sponsor. ? He suspected that PASS does not use the exact method (default in most function of PowerTOST ) but one of the approximations and tried the noncentral t (method = "nct" ) as well as the shifted central t (method = "shifted" ):
We know that in a 2×2×4 design power is approximately equal to a 2×2×2 design with ½ of its sample size because the number of treatments is the same and the differing degrees of freedom play a lesser role. In this case: 98 / 2 = 49 → 50. This approach is used in package bear .Since I’m not aware of reference tables for replicate design evaluated for ABE, I tried simulations (see this post for the code) and got for simulating statistics
What the heck? The output of PASS gives a list of references (I numbered the list):
#3 contains sample size tables and therefore, was a good candidate. Surprise: With increasing CV sample sizes were – generally – larger than expected. Unfortunately the tables don’t go beyond 40%. However, in Table VII 38 subjects are given, whereas I got 34. The underlying ABE-model is not specified; the authors refer to #1. OK, Chapter 9 is it. Gotcha, carryover in the model! Stephen Senn devoted a good part of his book about crossover studies arguing against it. Not only that carryover is scientifically questionable, none of the guidelines recommend such models. BTW, #1 contains also tables where the sample sizes are (consequently) too large. Given all that, I recommend to
PS: If you are with a CRO you might be tempted to sell the sponsor large studies. That might backfire like in this case where to sponsor knows PowerTOST …
— Dif-tor heh smusma 🖖🏼 Довге життя Україна! Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes |
ElMaestro ★★★ Denmark, 2020-05-15 15:01 (1670 d 04:07 ago) @ Helmut Posting: # 21441 Views: 6,355 |
|
Hi Hötzi, I could be wrong but... PASS may be assuming this is a case of two sample t-test-like-scenario, where it does not take into consideration that the actual model causes additional reduction of df's. You may be able to approximately reproduce PASS' result if you can override the df's in PowerTOST. I would still anyday say PowerTOST is more right than PASS, of course. — Pass or fail! ElMaestro |
Helmut ★★★ Vienna, Austria, 2020-05-15 15:42 (1670 d 03:26 ago) @ ElMaestro Posting: # 21442 Views: 6,406 |
|
Hi ElMaestro, ❝ I could be wrong but... We all may be but… ❝ PASS may be assuming this is a case of two sample t-test-like-scenario,… It does (according to the manual and given in the output). ❝ … where it does not take into consideration that the actual model causes additional reduction of df's. Yep, that’s the point! It gives me an answer to a question I did not ask (a model nobody uses for ages). Since I played around with the trial version, I can now say that it is a black ?. At least for the replicate designs the carryover model is hardcoded, not specified in the manual, and there is no way to change that. I just sent an email to NCSS’ support asking for a clarification. ❝ You may be able to approximately reproduce PASS' result if you can override the df's in PowerTOST. Maybe. ❝ I would still anyday say PowerTOST is more right than PASS, of course. I like this one: Of course, PASS passes IQ and OQ. Remember what you once wrote? — Dif-tor heh smusma 🖖🏼 Довге життя Україна! Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes |
ElMaestro ★★★ Denmark, 2020-05-15 21:02 (1669 d 22:07 ago) @ Helmut Posting: # 21443 Views: 6,300 |
|
Hi again, ❝ Yep, that’s the point! Let's face it: ElMaestro is always right. Except when he is wrong. You better get used to this very basic law of science. ❝ Remember what you once wrote? Hehe, it is a bit vague in my memory. Prions and loads of paint thinner have taken their toll on my brain. It must be inspired by something someone said, but I don't recall well. Perhaps from something I read in a Borland Delphi manual or something?! That would be the right place for a garbage discussion anyway. — Pass or fail! ElMaestro |
d_labes ★★★ Berlin, Germany, 2020-05-15 21:45 (1669 d 21:24 ago) @ Helmut Posting: # 21444 Views: 6,276 |
|
Dear Helmut, dear ElMaestro, ❝ ❝ I could be wrong but... ❝ ❝ We all may be but… Me too. Especially me . Just my 2 cents. Differences between PASS and PowerTOST w.r.t replicate cross-over designs
Don't use the sample sizes for replicate designs obtained by PASS because
— Regards, Detlew |
Helmut ★★★ Vienna, Austria, 2020-05-16 01:20 (1669 d 17:49 ago) @ d_labes Posting: # 21445 Views: 6,424 |
|
Dear Detlew, your first three points explain why sample sizes are generally larger. The last why differences increase with the CV. CV SE — Dif-tor heh smusma 🖖🏼 Довге життя Україна! Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes |
Helmut ★★★ Vienna, Austria, 2020-05-17 16:34 (1668 d 02:35 ago) @ Helmut Posting: # 21447 Views: 6,235 |
|
Dear all, due to Detlew’s detective work (THX!) I see it clearer now. Seems that it was a bug in earlier versions. “Equivalence Tests for the Ratio of Two Means in a Higher-Order Cross-Over Design (Log-Normal Data)” were added in PASS v14. Although nothing is stated about an improvement/update in later versions, according to the online manual (identical to the one which came with the PASS 2020 Trial) I could reproduce the examples with the internal function power.PASS() of PowerTOST . Code upon request.All examples for ABE {0.8000|1.2500}, α 0.05. Example 1 – Finding Power: ABB|BAA (Manual 545-9) The default in power.PASS() is the approximation by the shifted central t-distribution (method = "shifted" ), although the noncentral t (method = "nct" ) and the exact method by Owen’s Q (method = "exact" ) are implemented as well. Columns starting with PT give results obtained by power.TOST() .Confirmed that PASS uses the shifted t, which I solely used in the other examples. Good agreement with power.TOST() .Example 2 – Finding Sample Size: ABB|BAA (Manual 545-11) Good agreement (N1) though in practice one would round up to N2 in order to get balanced sequences like in all sample size-functions of PowerTOST .Example 3 – Validation using Chen et al. (1997): AA|BB|AB|BA (Manual 545-12) Good agreement again. Note that in order to reproduce the results of Chen et al. – despite CV is stated in the paper – we have to work with the standard error of residuals. Here it is with 0.1002505 close to the CV of 0.1 but see also there. Now the troublesome one of the OP. Example 4; PASS 15.05.5: ABBA|BAAB (SE instead of CV) I could not reproduce it exactly (the different design constants and dfs due to carry-over cut also in) but it explains what is going on in this earlier version of PASS and the discrepancy to sampleN.TOST() .Now what we can expect* in PASS Example 5 = 4; PASS 2000 (use CV) Seemingly OK. Conclusion: If you use PASS, update to PowerTOST ).
— Dif-tor heh smusma 🖖🏼 Довге життя Україна! Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes |
d_labes ★★★ Berlin, Germany, 2020-05-17 19:01 (1668 d 00:07 ago) @ Helmut Posting: # 21448 Views: 6,174 |
|
Really PASS 2000 or PASS 2020? — Regards, Detlew |
Helmut ★★★ Vienna, Austria, 2020-05-17 19:34 (1667 d 23:34 ago) @ d_labes Posting: # 21449 Views: 6,164 |
|
Dear Detlew, ❝ Really PASS 2000 or PASS 2020? F**k! 2020, of course. Freudian? IIRC, I once had Chris Hintze’s “NCSS 2000”. — Dif-tor heh smusma 🖖🏼 Довге життя Україна! Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes |
Helmut ★★★ Vienna, Austria, 2020-05-21 20:43 (1663 d 22:26 ago) @ Helmut Posting: # 21456 Views: 5,958 |
|
Dear all, in the following my observations/conclusion about PASS2020 v20.0.1 (released 2020-02-10). I checked only the sample size procedures relevant for ABE. CV 0.1–0.4 (Δ 0.02), 0.5, 0.75, 1.0; θ0 0.85–1.00 (Δ 0.05); AR {0.8000|1.2500}; target power 0.8 and 0.9. I compared the results of PASS with the exact method of PowerTOST and the SAS-code for the noncentral t-distribution given by Jones & Kenward (2000) ported to R. Not surprisingly in all of my 1,152 scenarios the exact method agreed with the noncentral t. PASS not so much… Paired samples The design for ratios is not directly accessible in PASS (only for differences). Novices (aka “Push-the-button statisticians”) might not know how to set it up based on logs and conclude that is not possible. 2×2×2 Accessible twice. Under the θ0 0.85, CV 1, power 0.8, I got in the former 2,334 (unless I ask for the exact sample size) and in the latter only 2,333. Likely most people use the latter and round up to next even number to get balanced sequences. Looks stupid if the output is part of the SAP. and . OK, why not. However, the results differ: For2×2×4 (TRTR|RTRT, TRRT|RTTR, TTRR|RRTT) Accessible under the
Terminology again. Following Chen et al. only the “Three-Period, Two-Sequence Dual ABB|BAA” given. Sigh. More or less OK. Following the EMA’s Q&A and assessing only studies with at least 24 subjects: x̃ ±0%, range –0.65 to +7.69%. 2×4×4 (TRTR|RTRT|TRRT|RTTR, TRRT|RTTR|TTRR|RRTT) Generally OK. Assessing only scenarios with n≥12: x̃ ±0%, range ±0% to +33.3%. 2×4×2 (TR|RT|TT|RR, Balaam’s design) Generally OK. Assessing only scenarios with n≥24: x̃ ±0%, range –0.54% to +5.26%. Higher-Order Designs (Latin Squares and Williams’ designs) for ratios are not implemented. Reference-scaling not implemented. We must no forget that any interventional trial carries some degree of risk. Hence, ICH E9 „Statistical Principles for Clinical Trials” stated already in 1998 The number of subjects in a clinical trial should always be large enough to provide a reliable answer to the questions addressed. Large enough, not larger…— Dif-tor heh smusma 🖖🏼 Довге життя Україна! Helmut Schütz The quality of responses received is directly proportional to the quality of the question asked. 🚮 Science Quotes |
mittyri ★★ Russia, 2020-05-25 00:12 (1660 d 18:56 ago) @ Helmut Posting: # 21459 Views: 5,739 |
|
Dear Helmut, I would give you one month NCSS business analyst salary if I were NCSS top manager. But I am not, sorry what I've found on their marketing page regarding customer satisfaction: PASS 13 is fantastic! Better than my new dishwasher and microwave combined. Never ever buy kitchen combines! — Kind regards, Mittyri |