A test count on its own is not evidence. What matters is which results are checked, against what, and to what tolerance — and where no external check exists at all.
This page is the answer to the question an HTA or regulatory reviewer actually asks. The comparator table below is generated from the suite in the merlin for R repository; the surrounding description is written against it, and the section at the foot sets out the limits, because a validation statement that lists only strengths is not one a reviewer can use.
survival · rstpm2 · flexsurv
agreement · truth recovery
checked against each other
Other people’s implementations
Where another package fits the same model, merlin is required to agree with it. These are the packages whose estimates the suite compares against, and the count is how many times each is called as a reference.
| Comparator | References | What is compared |
|---|---|---|
survival::survreg | 24 | Standard parametric AFT fits — the AFT trio (lognormal, loglogistic, generalised gamma), and the Royston–Parmar family at K=2, where it reduces to a Weibull and must recover survreg's answer. |
rstpm2::stpm2 | 19 | The closest existing R implementation on the Royston–Parmar log-cumulative-hazard axis. Matching its fixed-effect coefficient and log-likelihood is part of the definition of done for that family. |
flexsurv::flexsurvreg | 13 | Standard parametric PH and AFT fits — exponential and Weibull — checked through the prediction surface, not just the coefficients. |
flexsurv::flexsurvspline | 7 | Spline baselines on both the log-cumulative-hazard and log-hazard scales, against family = "rp" and family = "logh". |
survival::Surv appears far more often than any of
these, and is deliberately not counted here. It builds the response object; it does
not provide an answer to check against.
Three kinds of check
Agreement with another package
Fitted coefficients, log-likelihoods and predictions compared against the packages above, mostly at a tolerance of 1e-3 to 5e-3 — loose enough to absorb a different optimiser, tight enough that a real disagreement fails.
Analytic derivatives against numerical
merlin carries hand-derived first and second derivatives for every family, integration method and random-effect configuration. A dedicated set of test files differentiates the likelihood numerically and compares the two. This is the check with no external comparator, and it matters most: a subtly wrong gradient still converges, just to the wrong place.
Exact identity where required
Files asserting exact agreement, or agreement at 1e-10. The relative-survival kernels are the clearest case: with the excess-hazard term switched off they must be bit-identical to the ordinary survival path, so adding relative survival cannot perturb a result that never asked for it.
What this does not cover
- Multilevel models have no external comparator. Every cross-package test fits a single-level model with no random effects, because none of the comparators implement the multilevel case. The multilevel paths are checked against analytic derivatives, internal consistency and simulation — not against another package, because there is not one to check against.
- Comparator versions are not pinned. The tests run against whatever version of flexsurv, rstpm2 or survival is installed, and skip entirely if it is absent. A green run therefore says the comparison passed on that machine, on that day, against those versions.
- The recorded run is not a clean sheet. It carries warnings, and it skips tests — including whole files, where a comparator package is absent. A suite that skipped a check has not performed it, so the project’s own release record does not treat a run with skips as release-grade. Read the record in the repository rather than a figure reproduced here, which is a count from one run on one machine on one day.
- Passing tests are not a qualification. This is the evidence base, not a certificate. Where a submission needs a signed, version-locked release with validation documentation attached, that is a separate piece of work — see Validated.
Running it yourself
Nothing here needs to be taken on trust. The suite is in the package repository and runs in one line:
# from the merlin for R repository root
devtools::test()
# or, for a single area
testthat::test_file("tests/testthat/test-predict-vs-reference.R")If a number on this page does not reproduce for you, we would like to know — tell us.