Every pipeline drawing on more than one source eventually has to decide whether two records describe the same thing. A building footprint from one provider and a footprint from another. A road from an authoritative national dataset and the same road from a volunteer map. A business listing from an official register and a point of interest from an open dataset.
That decision is conflation, and it is where multi-source pipelines break in ways hard to see from the outside. The output looks like a single clean dataset, presented as better than either input because it has more coverage than one and more detail than the other. What is not visible is how many of its records are hybrids that neither source would recognise.
The failure is not that conflation is difficult in principle. It is that a mediocre conflation produces something that looks better than its inputs while being less trustworthy than either.
What conflation actually is
Conflation takes two or more datasets describing overlapping parts of the same reality and produces one in which each real world thing appears once, with the best available information about it and a record of where that information came from.
It decomposes into four operations worth keeping separate in the code, because conflating them is itself a source of bugs:
- Matching. Deciding which records in source A correspond to which in source B.
- Merging geometry. Deciding the shape of the output object when the sources disagree about where it is.
- Merging attributes. Deciding what name, classification, height, or status the output carries.
- Recording provenance. Preserving, for every output field, which source it came from and how the decision was made.
Most implementations do the first two, treat the third as a precedence list, and skip the fourth. The skipped one is what makes the result auditable.
Why every multi-source pipeline needs it
Nobody sets out to build a conflation engine. They arrive at one because no single source suffices.
Authoritative national datasets tend to be geometrically precise, legally meaningful, and slow. They have the road centreline surveyed properly and do not know it has been closed for six months. Volunteer maps are fast, rich in attributes that matter on the ground, and uneven in coverage. Machine-extracted datasets have consistent global coverage and consistent global errors. Commercial listings have business names and opening hours and go stale invisibly.
Any product needing both precision and freshness, or both coverage and attribute depth, has to combine sources. The moment two sources cover the same street, the pipeline is doing conflation, whether or not anyone has named it. Doing it implicitly, by loading one source after another into the same table and hoping the duplicates are obvious, is the most common approach and the worst.
Matching strategies, and why each one fails alone
There are three families of matching signal, and each has a characteristic failure that the others do not share.
Geometric proximity is the most obvious. Two objects occupying nearly the same space are probably the same object. For polygons this becomes overlap area, usually intersection over union. For lines it becomes buffer overlap or a directed distance. For points it is plain distance.
Geometry fails when precision differs between sources. A point-of-interest dataset that geocodes to the centre of a building and another that geocodes to the entrance place the same shop several metres apart. A footprint traced from low-resolution imagery and one surveyed from the ground differ by more than the spacing between adjacent terraced houses. Geometry also fails silently when one source carries a systematic offset, because a whole dataset can be shifted by a datum error and still be internally consistent.
Name similarity is the second family: string comparison, usually fuzzy, sometimes with token-level matching or phonetic normalisation.
Names fail in every direction. The same business appears as a trading name in one source and a legal entity name in another. Abbreviations, transliterations, and script differences break naive comparison. Chains defeat name matching entirely, since a dense retail district may hold four branches of the same operator within a few hundred metres, all identically named, so name similarity is maximal for every wrong pairing. Conventions vary too, so the same street is Jalan Sudirman in one source and Jl. Sudirman in another.
Attribute agreement is the third: house number, postcode, classification, height band, operator, opening date. Attributes fail because they are sparse. In most real datasets the discriminating attributes are populated for a minority of records, and those records are not a random sample: they are the well-mapped, urban, recently surveyed ones. Matching that depends on attributes therefore degrades exactly where data is thinnest, which is where the merge matters most.
The conclusion is not that one of these is best. It is that a matcher built on a single signal has a blind spot with a known shape, and combining signals covers the blind spots. A composite score with several weighted components, where any single component can be absent without failing the match, handles the real world far better than a cascade of hard rules. What it costs is interpretability, which is why the score components should be retained rather than collapsed to one number.
The dense-area failure
Nearest-neighbour matching works beautifully in the test area, which is usually somewhere the developer knows, and usually not dense.
Consider a row of terraced houses. Twenty properties, each about five metres wide, in a continuous row. Source A has them from an authoritative address dataset, positioned accurately. Source B has them from a different survey with a systematic offset of four metres along the row, well within the precision anyone would consider acceptable for a building dataset.
Nearest-neighbour matching now assigns every property in A to its neighbour in B. Every match is wrong, and every match is confident, because the residual distance is under a metre. The result is twenty buildings each carrying the attributes of the house next door: wrong addresses, wrong owners, wrong heights, in a dataset reporting twenty successful matches and zero failures.
This is not exotic. The same happens in market stalls, stacked apartment units, strip malls, and container yards, wherever object spacing is comparable with the positional uncertainty of the sources.
Three defences work in combination. Solve the assignment globally rather than greedily, so the matcher optimises total agreement across the set instead of committing to a first-come decision. Detect and correct systematic offsets before matching: if every candidate pair in a neighbourhood is displaced in the same direction by the same amount, that is a registration problem, not twenty coincidences. And refuse to match when the best candidate is not clearly better than the second best. A margin test comparing the top score against the runner-up converts a silent wrong match into an explicit ambiguity a later stage can handle.
One to many, many to many, and the terrace problem
The assumption that matching is one to one is wrong often enough to be dangerous.
A terrace row mapped as one polygon in a source that traces roof outlines corresponds to six houses in a source built from addresses. Neither is wrong. They describe the same fabric at different granularity, because a roofline and a legal dwelling are different objects occupying the same footprint.
The cases multiply. A shopping centre is one building and forty businesses. A dual carriageway is one road in one source and two carriageways in another. An apartment tower is one footprint and two hundred addresses stacked vertically, which no two-dimensional match resolves.
Handling this requires the schema to support cardinality other than one to one, which most do not. The output object needs to say it corresponds to one record in source A and six in source B, and to represent the relationship type: aggregation, subdivision, or genuine ambiguity.
Where the schema cannot express it, the honest fallback is to leave the objects separate and link them rather than force a merge. A pipeline emitting six house records and one building record with an explicit containment relationship is more useful than one emitting either six records carrying the building's roof geometry or one record with six conflicting addresses.
The specific trap is attribute smearing. When a one-to-many relationship is forced through a one-to-one merge, the attributes of the single record get copied onto all the many. Every house in the terrace inherits the roof area of the whole terrace, or the construction date of the block, or the ownership of the freeholder. The numbers are now wrong in a way that is arithmetically consistent, so totals look correct while every individual value is inflated.
What to do when the sources disagree
Once two records are matched, they will disagree about something. There are three responses, and the common default is the weakest of the three.
Pick a winner. Establish a precedence order, per field, and take the value from the highest-precedence source that has one. This is simple, fast, and produces a clean single-valued output. It suits fields where one source is genuinely authoritative: a national address register really is the authority on the address.
Its weakness is that precedence is global and correctness is local. The authoritative source is authoritative on average, not everywhere, and it is often the stale one. A rule that always prefers the official classification discards the volunteer observation that the bridge has a weight limit, which is the fact that matters to the lorry.
Keep both. Emit multiple values for the contested field, tagged by source. This preserves information and works well for attributes that are genuinely multi-valued, such as names in different languages. Its weakness is that it moves the problem downstream, and most consumers take the first value in the array without reading the tags.
Record the disagreement. Emit a resolved value, chosen by whatever rule applies, and alongside it a flag saying the sources conflicted, what each said, and by how much. The resolved value keeps the output usable by consumers who just need a number. The conflict record makes it auditable, and turns disagreement into a signal instead of a loss.
This third option is usually right, for a reason that is not obvious at first. Disagreement between independent sources is itself information. A building where two sources agree on the footprint to within a metre is a building we know well. A building where they disagree by fifteen metres is either badly mapped or genuinely ambiguous, and either way it should not drive a decision that depends on precision. Discard the conflict at merge time and that distinction is destroyed: every record looks equally reliable. Conflict rate is one of the few quality signals a multi-source pipeline gets for free, and a conflict flag also gives a review queue something to sort on.
Provenance has to survive the merge
Provenance is not a nice-to-have field. In a multi-source pipeline it is the mechanism by which the output remains legally and epistemically usable.
Licensing is the blunt reason. Sources carry different obligations, share-alike terms attach to derived works, and some permit commercial use while others do not. If the merged record cannot say which field came from which source, the pipeline cannot answer whether the output can be redistributed, and the safe answer becomes the most restrictive licence in the mix applied to everything.
The epistemic reason matters just as much. A user who sees a building height of eighteen metres wants to know whether that came from a lidar-derived surface model, from a floor count multiplied by an assumed storey height, or from a classification default. Those three deserve very different treatment and they look identical unless provenance is carried at field level.
Field-level provenance is more work than record-level and it is the level actually needed, because merged records are precisely the ones whose fields come from different places. The minimum useful payload per field is the source identifier, the source's own timestamp, and the method by which the value entered: taken directly, chosen by precedence over a conflict, derived, or defaulted. That last one matters most. A defaulted value that looks observed is the most dangerous thing a conflation pipeline can emit, because it will be counted and averaged alongside real measurements.
The confident hybrid
The characteristic failure of conflation is not a missed match or a wrong match. It is a record that matches neither source and carries the confidence of both.
It is assembled a field at a time, each step defensible. Geometry comes from the precise source. Name comes from the source with better attribute coverage. Classification comes from the authoritative source. Height comes from the one with heights. Each choice follows the precedence rules and each is individually reasonable.
The result describes an object that does not exist. The geometry is the old building, surveyed accurately before demolition. The name is the new business occupying the replacement. The classification is from a scheme predating the change of use. The height is from the demolished structure. Nothing is flagged, because no single field failed a check, and the record looks better than either input because it is more complete than both.
Two habits reduce this. Prefer coherent field groups over field-by-field precedence: take geometry, height, and structural attributes from one source as a bundle, so shape and measurements describe the same physical object. And run a consistency check across the merged record, so that if the geometry source and the attribute source disagree by more than a threshold about anything they both describe, the record is flagged rather than published quietly.
Neither habit is expensive. Both are usually skipped, because they generate flags, and flags require somebody to look at them.
The rule that keeps a merge honest
Before a conflated dataset ships, one property is worth testing directly. Take a sample of merged records and try to reconstruct, from the record alone, which source each field came from and what the alternatives were.
If that is possible, the merge is a defensible product. Consumers can decide for themselves, licensing can be answered, and errors can be traced back to their source.
If it is not, what has been built is a third dataset with no provenance, no error model, and no way to attribute a mistake to its origin. It will be more complete than either input and less trustworthy than both, and the completeness is what will get it adopted.