How we know it works
Every module was built by one AI agent and checked by a second, independent one, against known answers.
The short version
Every module passes the acceptance gate at its default design. Problems that were not fixed are written into each module's validation record as known limits, with numbers: for example, “at 100 sites the occupancy slope is biased +11.5%”. The agent reads those limits and warns you when your design is near one.
What “validated” means here: the method got known true values back from simulated data, at a stated design, within fixed limits. It does not mean the method is right for every data set, or that your data meet the model's assumptions. That is why every analysis repeats the check at your design.
How each module was built and checked
-
A builder agent writes the module
From a written specification and the primary literature, one agent writes the model, a simulator that generates data from that model, the report text and the figures. It then runs the planted-truth check at a default design and at harder designs, and records the results in the module's validation file.
-
An independent verifier tries to prove it wrong
A second agent, told to assume the builder's claims may be wrong, then:
- checks the literature: reads the original papers and package documentation and compares the equations, priors and derived quantities in the code line by line;
- writes its own simulator: builds new simulated data without reusing the builder's code, at a different design and different true values, fits 100 or more data sets and checks bias and coverage (bugs in a simulator and a model can cancel out, so the second simulator matters);
- runs negative controls: confirms the check can fail (a deliberately wrong model must fail it) and that an effect of zero is declared significant about 5% of the time, as it should be;
- cross-checks against established packages such as unmarked, secr, spOccupancy, lavaan and vegan, or against a second engine, on the same data;
- tests usability: feeds in messy data (missing values, a misspelled site ID, a wrong column name) and checks that the error messages are clear, and reads every sentence of the report for correct numbers and wording.
The verifier fixes real problems, re-runs the affected checks, and adds an “Independent verification” section to the module's validation record.
-
A worked example is run end to end
Each module has a realistic, simulated example study with a researcher's question and prompt. The full workflow is run on it, including planted truth at the example's own design. These are the examples on this site and in Appendix S1.
What the acceptance gate requires
An analysis is called DONE only when all of these hold. The thresholds are fixed in the code and are never loosened to make a case pass.
-
Plan first
What will be estimated, with which model and which checks, is written down before your data are fitted.
-
Planted truth at your design
At least 100 simulated data sets (50 if each fit takes minutes) with your design. For every quantity that will be interpreted:
- bias: the average estimate is within 10% of the truth (or within 3 Monte Carlo standard errors, whichever is larger);
- coverage: the 95% intervals contain the truth at least 95% of the time, minus 3 Monte Carlo standard errors (never below 80%);
- convergence: at least 90% of the simulated fits converge.
Why a quantity with a larger % bias can pass while one with a smaller % bias fails
The bias allowance for each quantity is the larger of two numbers: 10% of its true value, or 3 Monte Carlo standard errors. The Monte Carlo standard error is the uncertainty in the average estimate that comes only from running a finite number of simulations (it shrinks as more data sets are simulated). When the true value is small, 10% of it is tiny, and the noise floor sets the allowance instead, which can then be much wider than 10%.
For example, in the abundance-mediated interactions example (100 simulated data sets), the broadleaf effect on occupancy has truth 0.3 and average bias +0.065 (+21.8%). Ten percent of the truth is 0.030, but 3 Monte Carlo SE is 0.070, so the allowance is 0.070 and it passes. The marten-abundance effect has truth −0.6 and bias −0.066 (−11.0%). Its allowance is the larger of 0.060 and 0.063, and 0.066 is over it, so it fails. Both are about equally biased in absolute terms, and both sit close to the line.
So a PASS that relies on the noise floor means the bias could not be told apart from simulation noise. It does not mean the quantity is estimated well. Every module page now shows each row's allowance († when it comes from the noise floor) and marks ≈ borderline results: bias using 80–125% of its allowance, or coverage within 0.015 of the minimum. Such calls could flip with another set of simulations. Treat them as “about at the limit”, and read the size of the bias, not only the verdict.
-
Your fit converged
Bayesian models: R-hat below 1.01 and an effective sample size of at least 400 for every reported quantity, with trace plots inspected. Likelihood models: the optimiser converged and no estimate sits on a boundary.
-
The model fits your data
A goodness-of-fit check suited to the model, such as residual checks, MacKenzie-Bailey tests, posterior predictive checks or d-separation tests. A failed check is reported, not hidden.
-
Reproducible, and limits stated
Every number in the report comes from a script that can be re-run. Limits that apply to your design are stated in the report.
You still get the results, but marked OPEN, with the reason and what can and cannot be trusted. For example, the population time-series example answers “is the herd declining?” but not “is growth density dependent?”, because the density-dependence estimate was biased by -59% at that design.
The one critical error
The open-population spatial capture-recapture module (scr-open) estimates abundance, survival and recruitment over several years. Its builder gave each year's entry probability its own “flat” Uniform(0, 1) prior. That looks harmless, but together those priors strongly favour the idea that almost every possible animal entered the population at some point, so estimates are pulled upward whenever the data are weak.
The builder's own checks passed. The verifier's independent simulator, at a different design, found the problem: abundance was biased upward by 16 to 20% (superpopulation +19%, interval coverage 0.86), and the effect of elevation on density was 23% too strong (-0.86 against a true -0.70). Only 10 of 23 quantities passed. A maximum-likelihood fit of the same model was close to the truth, which showed the likelihood was right and the prior was to blame.
The fix was the superpopulation parameterisation of Kéry & Schaub (2012): one parameter for the share of animals ever alive, and entry probabilities that sum to one. With it, 23 of 23 quantities passed (superpopulation +8.1%, coverage 0.95; elevation effect -9.5%, coverage 0.91).
Two lessons. Reasonable-looking choices can bias results by 20% without any error message. And a check is only as good as its independence: the builder's tests could not see a problem the builder's own assumptions created.
What DONE and OPEN mean
Every worked example, and every analysis you run, ends with one of two verdicts.
DONE: every quantity the report interprets passed the planted-truth check at that design. Estimates were close to the truth on average, the intervals contained the truth about as often as they claim, and the fits converged.
OPEN: at least one quantity the report interprets did not pass at that design, so CascadiaTools does not certify it. It does not mean the module is broken: all 29 modules pass at their default designs. And it does not mean everything is wrong: usually most quantities pass and one falls short, often narrowly (for example, an interval that contains the truth 90% of the time when it should be 95%). The module page shows exactly which quantity, by how much, and why.
If your own analysis comes back OPEN, CascadiaTools tells you what you can still use and suggests what would help: more data (more sites, surveys, animals or years), a simpler model, a different estimator, or reading that quantity with extra caution (for example, treating a 95% interval as roughly 90%).
9 of the 29 worked examples are OPEN at their example designs. We kept them that way on purpose: they show the harness reporting a limit honestly, which is the point of checking.
Every worked example, with its verdict
Planted truth at each example's own design. “Rows passing” counts every quantity checked, including ones the report does not interpret (such as nuisance variances), so a DONE example can still have failing rows. The module page explains each one.
| Module | Worked example | Verdict | Rows passing | Borderline rows | Example design |
|---|---|---|---|---|---|
| GLM / GLMM with biological interpretation | Wood Thrush point counts: negative-binomial GLMM with survey-effort offset and station random intercepts… | ✓ DONE | 10 / 12 | 4 | differs from validated |
| Causal effects from observational data (DAG + g-computation) | Does riparian canopy cover cause more juvenile trout? Causal effect from 140 observational stream reaches… | ✓ DONE | 6 / 6 | – | larger than validated |
| Structural equation models (piecewise SEM and lavaan) | Fire severity, vegetation and songbirds after a wildfire (piecewise SEM with a Poisson GLMM… | ! OPEN | 18 / 24 | – | differs from validated |
| Single-species occupancy (static single-season and dynamic multi-season) | Pileated woodpecker occupancy and forest cover (simulated call-playback surveys, 120 stands) | ✓ DONE | 13 / 13 | 1 | smaller than validated |
| Multispecies occupancy with species interactions (Rota et al. 2016) | Do gray foxes avoid coyotes, and does housing density change that? Multispecies occupancy with species… | ✓ DONE | 38 / 38 | 1 | smaller than validated |
| Two-species conditional occupancy (dominant/subordinate; Richmond et al. 2010, Waddle et al. 2010) | Do invasive Cuban treefrogs push out native green treefrogs? Two-species conditional occupancy from PVC-pipe… | ✓ DONE | 18 / 18 | – | differs from validated |
| Spatial, multi-species and integrated occupancy (spOccupancy) | Fisher occupancy, canopy cover and elevation with residual spatial clustering (simulated camera-trap survey… | ✓ DONE | 7 / 9 | 1 | harder than validated |
| Abundance-mediated species interactions (dominant abundance -> subordinate occupancy) | Abundance-mediated species interactions: pine marten abundance and red squirrel occupancy (simulated… | ! OPEN | 11 / 12 | 4 | smaller than validated |
| N-mixture models for repeated counts (closed and open) | Hermit warbler abundance from repeated point counts (static binomial N-mixture model) | ✓ DONE | 8 / 8 | – | smaller than validated |
| Distance sampling (conventional, multiple-covariate and hierarchical) | Mule deer density on a winter range from line-transect distance sampling… | ! OPEN | 11 / 13 | 1 | same as validated |
| Spatial capture-recapture (closed population density) | Bobcat density from camera traps: closed spatial capture-recapture with a canopy-cover density covariate… | ✓ DONE | 7 / 7 | – | smaller than validated |
| Spatial mark-resight (SMR) and generalized spatial mark-resight (gSMR) | Fisher density from ear tags and trail cameras: generalized spatial mark-resight (simulated data) | ✓ DONE | 10 / 10 | – | same as validated |
| Search-encounter SCR (scat-detection dogs, snow-tracking, transect searches for genetic samples) | Fisher density from a scat-detection-dog survey with GPS-logged search routes (search-encounter SCR… | ✓ DONE | 8 / 8 | 1 | same as validated |
| Open-population spatial capture-recapture (Jolly-Seber SCR) | Pacific marten abundance, survival, recruitment and canopy association from five years of DNA hair snares… | ✓ DONE | 28 / 28 | – | same as validated |
| Cormack-Jolly-Seber apparent survival | Adult song sparrow apparent survival from constant-effort banding (Cormack-Jolly-Seber… | ✓ DONE | 10 / 10 | 1 | larger than validated |
| Multistate and multievent capture-recapture (Arnason-Schwarz / Pradel multievent) | Chytrid (Bd) infection, survival and clearance in a PIT-tagged frog population (multievent capture-recapture… | ✓ DONE | 16 / 16 | 1 | same as validated |
| Jolly-Seber open-population capture-recapture (abundance, survival, recruitment) | Western pond turtle abundance, recruitment and survival from eight years of PIT-tag hoop-trapping… | ✓ DONE | 42 / 45 | 1 | smaller than validated |
| Integrated population model (counts + capture-recapture + productivity) | Red-backed shrike population dynamics from counts, ringing and nest monitoring (integrated population model… | ✓ DONE | 19 / 25 | 2 | same as validated |
| Population time series - trends, density dependence and drivers of growth (state-space models) | Is a mule deer herd declining, and do deep-snow winters reduce its growth? (Bayesian state-space model… | ! OPEN | 5 / 8 | 1 | larger than validated |
| Population viability analysis - count-based (diffusion / state-space) and matrix-model PVA | Shorebird breeding-pair survey: count-based state-space PVA with counting error and missing years (simulated) | ! OPEN | 6 / 7 | – | harder than validated |
| Species distribution from presence-only records (point process, Maxent, integrated with surveys) | Pacific marten presence-only records: point-process species distribution model with road (observer-effort)… | ✓ DONE | 3 / 3 | – | same as validated |
| Joint species distribution model (HMSC) | Joint species distribution model (HMSC): forest breeding birds along temperature and canopy gradients… | ! OPEN | 12 / 17 | 2 | larger than validated |
| Community composition - ordination, RDA and variation partitioning, PERMANOVA with dispersion test, beta-diversity partitioning, indicator species | Forest bird communities along elevation and canopy gradients and among stand types (simulated Oregon Cascades… | ! OPEN | 10 / 11 | 1 | same as validated |
| Hidden Markov models for animal movement (behavioural states) | Behavioural states of GPS-collared elk with a hidden Markov model (simulated example) | ! OPEN | 15 / 16 | 2 | same as validated |
| Habitat selection from telemetry - integrated step selection (iSSA), RSF and home ranges | Habitat selection of GPS-collared elk: integrated step-selection analysis (iSSA)… | ✓ DONE | 10 / 11 | – | smaller than validated |
| Genetic diversity and population structure (FST, FIS, isolation by distance, DAPC) | Genetic diversity and population structure of coastal tailed frogs in eight Oregon Coast Range streams… | ✓ DONE | 7 / 7 | 1 | smaller than validated |
| Effective population size and relatedness | Effective population size and relatedness of a bull trout spawning population from two microsatellite samples… | ! OPEN | 3 / 5 | 1 | harder than validated |
| Fish growth (von Bertalanffy) and length-weight condition | Do female and male walleye differ in growth and condition? (simulated autumn 2014 gill-net survey) | ✓ DONE | 17 / 17 | – | larger than validated |
| Fish stock assessment - catch curves, stock-recruitment and surplus production | Walleye catch curve: total annual mortality from fall gill-net ages (simulated) | ✓ DONE | 5 / 7 | 1 | harder than validated |
All the details
Appendix S1 has every worked example in full: the study
design, true values, recovery tables, estimates against the truth, convergence and goodness of
fit, all figures, and the commands to reproduce them. Each module's complete validation record
(VALIDATION.md) is in the repository.
