Benchmark timings
Wall-clock times for merlin and for other R packages fitting the same model to the same data, reported next to the log-likelihood and coefficient each one returned.
The first thing to read here is the agreement, not the seconds.
On the Weibull frailty fit below, merlin and frailtypack return the same log-likelihood, the same coefficient and the same frailty standard deviation to every digit reported. On the 100,000-row Royston–Parmar fit, merlin and flexsurv return an identical log-likelihood and all three packages agree on the coefficient to four decimal places. Independent implementations, written by different people over more than a decade, converging on the same answer is the useful result. The timings are a practical matter downstream of it.
A note on the packages here
flexsurv, rstpm2, frailtypack, cuRe and flexsurvcure are long-standing, widely used packages, several of them implementing methods merlin also implements. They are the work of researchers whose papers we cite and whose software this field — ourselves included — has relied on for years. merlin was written knowing them, and in places because of them.
Where merlin is faster on these configurations, it is generally because it was written later and narrower: one compiled engine, analytic first and second derivatives, and a single likelihood path rather than a general-purpose one. That is an engineering difference on a particular fit, not a statement about the statistics, and it does not generalise to models or options these packages support and merlin does not.
Timings
Seconds are wall-clock on a single host, so read them as ratios within a row rather than as absolute performance. Log-likelihoods and coefficients are what each package returned on the same simulated data.
| Package | Method | Seconds | Log-likelihood | Coefficient |
|---|---|---|---|---|
| merlin | analytic, no RE | 0.64 | −83895.67 | 0.3024 |
| rstpm2 | stpm2 | 5.06 | −83896.15 | 0.3024 |
| flexsurv | spline | 23.78 | −83895.67 | 0.3024 |
| Package | Method | Seconds | Log-likelihood | Coefficient |
|---|---|---|---|---|
| merlin | AGHQ | 0.20 | −8775.48 | 0.2781 |
| merlin | Laplace | 0.47 | −8775.74 | 0.2779 |
| frailtypack | Weibull, log-normal frailty | 38.35 | −8775.48 | 0.2781 |
| Package | Method | Seconds | Log-likelihood | Coefficient |
|---|---|---|---|---|
| merlin | AGHQ | 1.66 | −42080.7 | 0.2913 |
| rstpm2 | frailty, log-normal, Q=7 | 11.27 | −42080.96 | 0.2913 |
| coxme | Cox frailty | 5.12 | not comparable | 0.2911 |
merlin for Stata, version to version
v3.0.0 rebuilt the likelihood engine with analytic gradients and Hessians. This compares it against v2.4.7 on identical scenarios, each version built from its own source. All ten scenarios are shown, not a selection: the gain ranges from almost nothing on simple fixed-effects fits to a different order of magnitude on the Cox path, which was re-implemented rather than tuned.
| Scenario | N | v2.4.7 (s) | v3.0.0 (s) | Same log-lik. |
|---|---|---|---|---|
| Weibull, 5 covariates | 500 | 0.201 | 0.178 | yes |
| Weibull, fixed effects | 200,000 | 0.699 | 0.588 | yes |
| Royston–Parmar, df 4, fixed effects | 200,000 | 1.065 | 0.932 | yes |
| Log-hazard splines (RCS) | 10,000 | 1.613 | 0.493 | yes |
| Weibull, time-dependent effect | 10,000 | 1.037 | 0.511 | yes |
| Cox, fixed effects | 10,000 | 19.461 | 0.042 | yes |
| Weibull, two-level random effects | 6,000 | 0.466 | 0.192 | yes |
| Gaussian, random intercept + slope | 3,000 | 1.931 | 0.656 | yes |
| Survival prediction, point estimate | 200,000 | 0.183 | 0.159 | — |
| Survival prediction with CIs | 200,000 | 6.621 | 1.692 | — |
How these were produced, and what they do not control
- Generated by
Rscript bench/compare.Rin the merlin for R repository, under R 4.6.0. The curated snapshot lives atbench/PERF-COMPARISON.mdwith per-section provenance and commit hashes. - One machine, one run per configuration. The benchmark's own notes record that the host thermally throttles under sustained load, which is why comparisons are made within a run and not across days.
- Comparator package versions are not pinned in the published snapshot. Timings will move with future releases of any package here, including merlin.
- Each configuration is simulated data chosen to exercise one model class. Nothing here measures a package on the models it was designed for and merlin is not.
- coxme fits a Cox frailty model, so its log-likelihood is not comparable with the parametric fits and is marked as such.
If a number here does not reproduce for you, we would like to know — tell us.