Validation of non­linear mixed-effects soft­ware [Software]

posted by Helmut Homepage – Vienna, Austria, 2015-12-30 20:56 (3011 d 13:16 ago) – Posting: # 15785
Views: 4,975

Dear all,

inspired by this thread some comments.
Nonlinear mixed-effect models are complicated. Really! Essentially the optimizer in the software tries to find the global minimum of the objective function (OF). Once the minimum is found, the coordinates are the “best” parameter estimates. Sounds easy. But: The OF is a hypersurface in an n+1-dimensional parameter-space (n = number of parameters in the model). To complicate things, coordinates of the n+1-space can be continuous (PK/PD parameters, covariates like weight, age, …), discrete (time to event, …), categorial (disease state, …), or even dichotomous (sex, …). Only for some basic PK/PD models closed solutions are possible. In all other cases partial derivatives have to be approximated numerically. Many clever algorithms exist: Parametric (Gauß-Newton, Newton-Raphson, Levenberg-Hartley, …), semiparametric (grid/simplex searches like Nelder-Mead), or nonparametric (Monte Carlo Markov Chains, …).

There are many traps set for our poor optimizer:
  1. The hypersurface is “flat”. There might be even a gradient towards a minimum but it is below the numeric resolution of the machine. The optimizer wanders around for a while (or hours) only to come to the conclusion that a minimum doesn’t exist. That’s the nasty no-convergence situation we all hate.
  2. Several minima exist which cannot be resolved at the numeric precision. Rare, but especially nasty. Which one to pick?
    Happened to me once. After months (!) of work my final result was this: Depending on starting values I got three solutions which were practically indistinguishable (p-values close to 0.9, AICs differed at the fourth digit). In technical terms, the model was not robust. The sponsor took it with good humor in the tradition of Edison:
            I have not failed 700 times. I have not failed once.
        I have succeeded in proving that those 700 ways will not work.

    If we are lucky we can rule some solutions out because their estimates are physiologically not plausible.
  3. With the given starting values a local minimum instead of the global one is reached. Our aim is the black hole in the center of the milky way only to find our spaceship sucked into the gravity well of Betelgeuse. See the wonderful example of the NIST.
There are many strategies to overcome these obstacles.After sleepless nights, Schützomycin, and a lot of trial an error you get something you are happy with. But wait! This is not a hobby project, you have to submit the stuff to an agency. Are there some guidelines to observe? Oops.While reading these gems you promise to yourself that the next time you will read them before you start modeling. So much to do: Visual prediction checks, bootstrapping, robustness, training and validation sets, internal and external validation, … You feel dizzy. Time for another pill.
OK. You slept over it. Learning curve. Next time you will do better. You read the GLs once more. Do they speak about cross-validation (i.e., comparing results obtained by different software)? Not a single word. That’s interesting (and related to the thread I mentioned in the beginning).
In 2005 Pascal Girard and France Mentré had a splendid idea. Generate data sets from a known PK model (one-compartment with first order absorption), and spice the data with error from a known distribution. Send the data sets to (very!) experienced pharmacometricians who would try to come up with estimates in different software. Initial starting values for fixed effects were provided. The outcome was surprising. Some participants would have used a different PK-model. In this case they were provided with the true one. See the original presentation1 and have a look at the bias of estimates.

Given that, I’m extremely skeptical whether it is possible to reproduce NLME-results of software A in software B.

The title of the first chapter of Peter Bonate’s excellent textbook2 is “The Art of Modeling”. All too true!


  1. Girard P, Mentré F. A comparison of estimation methods in nonlinear mixed effects models using a blind analysis. PAGE Meeting: Pamplona, Spain (16–17 June 2005). [image] free resource.
  2. Bonate PL. Pharmacokinetic-Pharmacodynamic Modeling and Simulation. New York, Springer: 2nd ed 2011. doi:10.1007/978-1-4419-9485-1.

Dif-tor heh smusma 🖖🏼 Довге життя Україна! [image]
Helmut Schütz
[image]

The quality of responses received is directly proportional to the quality of the question asked. 🚮
Science Quotes

Complete thread:

UA Flag
Activity
 Admin contact
22,957 posts in 4,819 threads, 1,636 registered users;
118 visitors (0 registered, 118 guests [including 6 identified bots]).
Forum time: 10:12 CET (Europe/Vienna)

With four parameters I can fit an elephant,
and with five I can make him wiggle his trunk.    John von Neumann

The Bioequivalence and Bioavailability Forum is hosted by
BEBAC Ing. Helmut Schütz
HTML5