A spatial model that has never been checked against the world is a hypothesis with a user interface. It emits building heights, road widths, canopy cover, population estimates, and every one of those numbers looks identical whether it is correct or badly wrong. Nothing in the output distinguishes them.
Validation is what separates the two. It is also the first thing cut from a schedule, because it is slow, it needs data nobody has budgeted for, and it usually makes the model look worse than the demo did.
So teams substitute something cheaper that feels like validation. Most of what gets called validation in spatial products is that substitute.
Internal consistency is not accuracy
The cheapest thing to measure about a model is how well it agrees with itself. Run it twice on the same tile and the outputs match. Run it on adjacent tiles and the values vary smoothly. Split the training data, hold out a fifth, and the held-out error is small. All of this is real information, and none of it is accuracy.
A confidently wrong model is extremely consistent. Suppose a building height estimator derives height from shadow length and carries a single wrong assumption about the sun angle model, or a systematically wrong ground elevation. Every output is off by roughly the same factor. Variance between runs is near zero. Adjacent tiles agree beautifully. The held-out split agrees too, because the held-out data came from the same source with the same bias baked in. Every internal check passes and every number is wrong.
This is the old distinction between precision and accuracy, and it matters more in spatial work than in most places because spatial errors are spatially correlated. Random noise would at least show up as roughness. A systematic error produces a smooth, plausible, wrong surface, and smoothness reads to most reviewers as quality.
Held-out validation deserves particular suspicion. Holding out a slice of the same dataset measures how well the model generalises within that dataset's own error structure. If the dataset has a coverage bias, a labelling convention, or an outdated survey date, the held-out slice inherits all three. The measured error is the error against the source, not against the world.
What ground truth actually means
A reference measurement earns the name ground truth only if it satisfies two conditions. It has to be produced by a process independent of the model, and it has to be more accurate than the model.
Independence is where most validation quietly fails. If a footprint model was trained on OpenStreetMap and is then evaluated against OpenStreetMap, that is not validation, it is a mirror. The subtler version is shared ancestry: two datasets that look independent because they come from different organisations, but which both ingested the same upstream import years ago. Checking provenance is not optional. Read the lineage of the reference before comparing anything to it.
The second condition is usually treated as absolute and never is. Reference data has its own error. A cadastre records the legal parcel, not the built structure. A field survey has instrument error and transcription error. Which means a comparison does not measure model error, it measures disagreement between two imperfect measurements. Attributing all of that disagreement to the model overstates the model's error when the reference is weak and understates it when the two share a bias.
The practical form of honesty here is to state the reference's own uncertainty alongside the result. If the reference is good to about half a metre and the measured disagreement is half a metre, the correct conclusion is that the comparison cannot resolve the model's error at that scale, not that the model is accurate to half a metre.
The validation ladder
There is a natural ordering of validation methods by cost, and a matching ordering by strength. The two run together, which is inconvenient.
- Cross-source agreement. Take two independent datasets covering the same quantity and compare them. No fieldwork, no budget, runs overnight. What it produces is a disagreement map: where the sources differ and by how much. What it cannot produce is a verdict on which one is right. Agreement is weak evidence of correctness, because two sources can be wrong the same way. Disagreement is strong evidence that something is wrong somewhere, which is genuinely useful as a screen for finding the places worth spending real money on.
- Held-out comparison against an independent reference. A dataset built by a different process, for a different purpose, that the model has never seen. A municipal cadastre, an engineering survey, a national statistical product. This yields an actual error distribution rather than a disagreement map. Its limitation is coverage: independent reference data exists overwhelmingly in places that are already well administered and well mapped, which are the places the model already handles.
- Field measurement. Someone goes to the location and measures the thing. This is the only rung that breaks the circle, because it introduces information that did not come from any existing dataset. It is expensive per sample, slow, and logistically awkward, so the sample count is always small. It also has failure modes of its own: instrument error, protocol drift between crews, and the strong tendency of field teams to sample where access is easy.
Each rung answers a stronger question at higher cost. The common failure is stopping at the first rung, seeing broad agreement, and describing the model as validated.
Sampling design, and why convenience samples flatter
Where the validation samples come from determines what the resulting number means, and convenience is a powerful force.
Validation samples tend to land where reference data is available, where imagery is clear, where the imagery is recent, where roads are wide enough to see, where buildings are rectangular, where a field crew can park. Every one of those conditions correlates with the model performing well. The result is an accuracy figure computed on the easy subset and reported as accuracy overall. Nothing in the process feels dishonest, and the number is still wrong.
The fix is stratification. Enumerate the variables that plausibly drive error: built density, structure type, terrain slope, image acquisition date, cloud and haze, vegetation season, informality of the settlement pattern. Define strata over those variables, sample within each stratum, and report error per stratum rather than pooled. If a single headline figure is required, compute it as a weighted average using the real-world frequency of each stratum, not the frequency in the sample.
Two details that get missed. First, small strata produce wide confidence intervals, and the honest move is to publish the interval rather than the point estimate, because a stratum with six samples does not have a meaningful mean. Second, spatial autocorrelation quietly destroys effective sample size: a hundred validation points drawn from one neighbourhood are not a hundred independent observations, because they share imagery, share a survey date, share a building vernacular. Spread samples geographically even when clustering would be cheaper, and treat the effective sample count as smaller than the raw count.
Systematic versus random error
The single highest-value distinction in validation is between error that averages out and error that does not.
Random error is scatter. Individual estimates land above and below the truth with no consistent direction. It shrinks when observations are aggregated, which means it matters most for single-object queries and least for area summaries.
Systematic error is bias. Every estimate leans the same way. It survives aggregation completely. Worse, it becomes proportionally more important as data is aggregated, because aggregation removes the random component and leaves the bias standing at full size. District totals, which is what people actually make decisions on, are precisely where bias is most dangerous and least visible.
A concrete shape of this: a canopy cover model that cannot resolve crowns below a certain diameter will miss small trees everywhere. Per-tree, the error looks modest and noisy. Aggregate to a neighbourhood and the miss becomes a consistent underestimate of total canopy, in exactly the neighbourhoods with young plantings, which are often the ones being assessed for a planting programme.
The operational consequence is that thirty carefully chosen samples that let a bias be estimated and removed are worth more than three hundred scattered spot checks that merely tighten a random-error band which aggregation was going to remove anyway. Validation budget should be spent on finding structure, not on accumulating counts.
Finding structure is mechanical: plot residuals against every covariate available. Bias appears as slope, as a step, as a curve. A residual plot that is flat across a variable is evidence the model is not being fooled by that variable, which is one of the few genuinely reassuring results in this work. Where structure appears, fit a correction, and then validate the correction on samples that were not used to fit it. Skipping that last step converts an overfitted model into an overfitted model with an extra calibration layer.
One accuracy number is a lie of aggregation
A model that outputs building height, road width, and canopy cover has three distinct error behaviours with three distinct causes. Height error is driven by roof geometry and elevation reference. Width error is driven by occlusion and parked vehicles. Canopy error is driven by season and shadow. Collapsing all of it into a single headline percentage destroys the information a consumer needs.
Report an error band per figure type, and within each type, per stratum. Report the shape of the distribution, not only its centre, because a mean absolute error hides a long tail and the tail is what produces the incidents. A useful minimum is three things per figure type: an interval that covers the bulk of cases, a description of the tail behaviour, and a failure rate meaning the fraction of inputs where the model produced nothing or produced something obviously nonsensical.
Choose the error unit to match the quantity. Relative error is right for quantities spanning orders of magnitude, absolute error for bounded ones. Ten percent error on a three metre structure and ten percent on a forty storey tower are not the same problem, and neither is well described by the other's convention.
Validate where the model is weak
The instinct is to validate where reference data exists. Reference data exists where the world is well administered. The world is well administered where the model already performs well. Following the instinct produces a number that describes the easy case and says nothing about the hard one.
The corrective is adversarial validation. Before collecting anything, write down the conditions expected to break the model: informal settlement patterns with non-rectangular footprints, streets under continuous tree cover, construction newer than the imagery, mixed-use buildings where the ground floor contradicts the structure, terrain with strong relief, monsoon-season acquisitions. Then go and buy or collect reference data specifically in those conditions, accepting a much higher cost per sample.
This feels wasteful because those samples produce the ugliest numbers. They are the samples worth having, because they are the only ones that reduce uncertainty about the error band. Adding a hundred more samples in a place where the band is already tight changes nothing anyone will act on.
The same logic applies at the edge of coverage, where the input data thins out. A model that degrades gracefully and says so is a different product from one that fails silently and keeps emitting confident numbers. Only validation in thin-data conditions can tell which one has been built.
Publishing the failures
A documented failure mode is more useful to a consumer than a headline accuracy, and considerably more useful to an automated consumer that has no intuition to fall back on.
What is worth publishing: per-stratum error rather than a pooled figure, an explicit list of strata where no reference data exists at all, known biases and whether they are corrected or merely known, the acquisition dates of both model inputs and reference data, and the conditions under which the model declines to answer. Where a stratum has never been validated, the correct entry is that it has never been validated, not an interpolated guess from neighbouring strata.
There is an incentive argument for doing this early. A limitation published before launch is a known constraint that consumers design around. The same limitation discovered by a customer after launch is a defect, and the conversation is entirely different. Nothing about the model changed between those two situations except who found it first.
The test that decides whether a figure ships
Before publishing any figure, there is one question worth asking about it: what measurement would demonstrate that this number is wrong, and has that measurement been attempted.
If no measurement could contradict it, it is not a measurement, it is an assertion with units. If a measurement exists and was not attempted, the figure ships with an unbounded error band and should be labelled that way. If it was attempted in the easy strata only, the band applies to the easy strata only, and saying so costs nothing except the appearance of confidence.
That is the whole discipline. Not achieving high accuracy, which depends mostly on the data available, but knowing the size and the shape of the error well enough that nobody downstream is surprised by it.