Two systems can both claim to find buildings in aerial imagery and mean entirely different things by it. One returns rectangles that contain buildings. The other returns, for every pixel, whether that pixel is building. The distinction sounds academic until a number computed from the output reaches a decision, at which point it becomes the difference between a usable measurement and a confident fiction.
Overhead imagery makes both harder than the ordinary photograph case that most computer vision intuition comes from. Objects sit at arbitrary rotations, scale changes with the sensor and the zoom level rather than with distance, and the notion of a canonical upright orientation does not exist. Models trained on street level photographs carry assumptions that do not survive the change of viewpoint.
What follows is the mechanical difference between the two approaches, the failure modes specific to looking straight down, and the part almost everyone skips: checking whether the output is true rather than admiring how crisp it looks.
A box and a mask answer different questions
A bounding box is four numbers. It says an object of this class is somewhere inside this rectangle. It carries no information about the object's outline, its orientation, or how much of the rectangle it occupies.
A mask is a per-pixel labelling. It says exactly which pixels belong to the object, so the object's shape and extent are recoverable directly.
For area, this is decisive. The area of the box is not the area of the building. For a rectangular building aligned with the image axes the two are close. For the same building rotated forty-five degrees the axis-aligned box has roughly twice the area of the structure inside it. Buildings in the real world sit at whatever angle the street grid or the plot boundary dictates, so a pipeline that computes footprint area from boxes returns a number whose error depends on the local street orientation. That error is not noise. It varies systematically between neighbourhoods with different grid alignments, which means comparisons between areas are corrupted in a way that looks like a real geographic pattern.
Anything derived from area inherits the same defect: floor area, coverage ratio, impervious surface, solar potential, material estimates. The mistake happens once, in the choice of output type, and then propagates silently through every figure that follows.
When a box is genuinely enough
Boxes are cheaper to produce, cheaper to store, cheaper to annotate and faster to run. Preferring masks reflexively wastes effort on tasks that never needed them.
A box is sufficient when the question is about presence, count or approximate location. Counting vehicles in a car park, checking whether a structure exists at a coordinate, finding candidate regions for a second stage of processing, monitoring whether the number of objects in an area has changed: none of these need an outline.
A box is also sufficient when objects are compact and roughly isotropic, since the box then approximates the extent well. Round storage tanks, small vehicles and isolated trees lose little from being represented as rectangles.
A mask is required whenever area, shape, adjacency or boundary position matters. Footprint extraction, land cover proportions, canopy coverage, water extent, anything that will be converted into a polygon and joined against other geometry. If the output is going to become a vector layer, it needed to be a mask.
An intermediate option worth knowing exists: the oriented box, which adds a rotation angle to the four numbers. For elongated objects at arbitrary angles, ships, aircraft, long agricultural structures, it recovers most of the area accuracy of a mask at a fraction of the cost. Overhead imagery is one of the few domains where oriented boxes are standard rather than exotic, precisely because rotation is unconstrained.
Instance versus semantic segmentation
Two things are both called segmentation and they are not interchangeable.
Semantic segmentation labels every pixel with a class and stops there. A semantic building layer tells you which pixels are building. It does not tell you where one building ends and the next begins. Run over a terrace of attached houses, it produces one connected region, correctly, because every pixel in it is genuinely building.
Instance segmentation separates objects. Each detected structure gets its own mask with its own identity. This is what is needed for any per-object question: counting, per-building attributes, joining to addresses.
Choosing semantic when instance was needed produces the merged terrace problem, and it produces it as an inevitability rather than as a model failure. No amount of accuracy improvement in a semantic model will split adjoining buildings, because the task as posed does not ask it to.
The usual patch is to run connected component labelling on the semantic output and call each component an instance. This works where buildings are detached and fails everywhere else, which is most urban fabric outside low density suburbs. Watershed and similar splitting heuristics improve it slightly and introduce their own arbitrary boundaries.
There is also a combined formulation, sometimes called panoptic, which labels every pixel and separates instances of the countable classes while treating continuous classes like road surface or vegetation as regions. For overhead work this is often the honest framing, since a scene contains both things that come in units and things that do not.
Failure modes specific to looking down
Overhead imagery violates several assumptions baked into general purpose vision models.
Rotation is unconstrained. In a street photograph, cars have wheels at the bottom and sky at the top. From above, a building can present at any angle, and there is no gravity cue to define which way is up. Models must either learn rotation invariance from data covering all orientations, which needs a lot of data, or have it built in through augmentation. Where this is done poorly, detection quality varies with orientation, and since orientation correlates with street grid, quality varies geographically.
Scale is set by ground sample distance, not by distance to the camera. Two images of the same city at different resolutions present the same building at different pixel sizes, and a model trained at one resolution can degrade sharply at another. Worse, the relationship is not simply about size: at coarser resolution, small structures do not merely shrink, they blend into their surroundings and disappear entirely.
Objects are small and numerous. A tile may contain hundreds of instances, many of them a handful of pixels across. Architectures designed for a few large objects per image handle this badly, and the standard evaluation habits designed for those architectures hide the problem.
Background dominates. In most overhead tiles the class of interest occupies a small fraction of pixels. This imbalance pushes models toward predicting background, which shows up as good overall accuracy and poor recall on the thing anyone cared about.
Tiling introduces its own artefacts. Large images must be cut into tiles for processing, and objects crossing a tile boundary get split, detected twice, or missed. Overlapping tiles with merge logic fixes most of it and creates duplicate handling problems of its own. Regular artefacts along a grid in the output are a signature of this, not of anything in the world.
Shadow and occlusion behave differently from ground level. Tall structures occlude their neighbours at off-nadir angles, and shadows are large, hard edged and directionally consistent across the scene. That consistency is exploitable, because a dark region whose direction from a structure disagrees with the sun azimuth is probably not a shadow.
The confidence threshold moves precision against recall
Every detector emits a score per detection, and a threshold turns scores into results. That threshold is a policy decision, not a technical detail, and it is usually left at whatever the example code set.
Raising it produces fewer, more reliable detections: precision rises, recall falls. Things that exist go missing. Lowering it catches more real objects and admits more false ones: recall rises, precision falls.
Which direction is correct depends entirely on what happens downstream. A screening pipeline where a human reviews every hit should run permissive, because a missed object never gets reviewed while a false one costs a few seconds. An automated pipeline whose output goes into a published figure should run conservative, because a false detection becomes a fact.
Two properties of the score are widely misunderstood. First, it is not a probability of correctness in any calibrated sense unless it has been explicitly calibrated, and most are not. A score of 0.9 does not mean nine out of ten such detections are right. Second, score distributions shift between regions. A threshold tuned on one city produces a different operating point on another with different building materials, imagery vintage or sun angle. A single global threshold delivers inconsistent behaviour across the map, and the inconsistency looks like real geographic variation in building density.
The defensible approach is to set the threshold per deployment region against a small local validation sample, and to record which threshold produced any given output layer. Without that record, two layers produced months apart cannot be compared at all.
The lattice failure
There is a specific and instructive failure that overhead detectors produce on repetitive texture: a regular grid of detections where nothing exists.
It appears over solar farms, greenhouse clusters, container yards, orchards, large parking areas, ploughed fields and regularly patterned roofs. The texture has periodic structure at roughly the scale the model associates with the target class, and the model responds at every period, producing a neat lattice of boxes or masks.
What makes it dangerous is that it does not look like an error. Random false positives look like noise and get noticed. A lattice looks like a discovery: a regular array of structures, evenly spaced, exactly what a planned development or an industrial site would produce. It passes visual inspection because human pattern recognition reads regularity as intent.
Detecting it needs an explicit check rather than a glance. Regularity in detection spacing is measurable: compute nearest-neighbour distances within a local window and look for a distribution far tighter than real settlements produce. Real building spacing is variable even in planned developments. Detections at a near constant pitch across a large area, especially with near identical confidence scores, are almost always model resonance with texture.
The related pathology is the response to linear features: model output following a road, a field boundary or a river as a chain of detections. Same cause, same tell, same remedy.
Validate the output, do not admire it
Modern segmentation output looks convincing. Boundaries are sharp, shapes are regularised into clean polygons, and the overlay on imagery is persuasive at the zoom level people review it at. None of that is evidence of correctness. Crispness is a property of the post-processing, and post-processing runs identically on correct and incorrect detections.
Useful validation has a few properties.
- Stratify the sample. Evaluate separately across dense core, suburban, informal, industrial, rural and any locally distinctive fabric. A single aggregate figure is dominated by whatever class covers the most area, which is rarely the class of interest.
- Hold out geography, not just images. Randomly splitting tiles from one city puts near identical scenes in both training and evaluation, and the resulting figure describes memorisation. The honest test is a city the model never saw.
- Measure the quantity that will be used. If the output feeds area calculations, evaluate area error, not just whether a detection overlaps a ground truth object. A mask can overlap correctly and be twenty percent too large.
- Count both directions of error separately. Missed objects and invented objects have different downstream consequences and different causes, and a combined score hides which one is happening.
- Look at the worst tiles, not the average. The average is reassuring and uninformative. The tiles where the model failed badly reveal the systematic cause, and systematic causes are fixable.
What to do with a detection layer once it exists
The last step is deciding what the layer is allowed to claim. A raw detection layer is evidence, not fact, and the useful discipline is to keep it labelled that way through the pipeline: each feature carrying its score, the threshold in force, the model version, the imagery date and the region-specific validation result that applies to it.
That metadata is what allows a later consumer to answer the only question that matters when a number is challenged: is this a building, or is it a place where a model responded strongly to something that looked like one? Without it, every detection in the layer has exactly the same standing, which means the reliable ones have been dragged down to the level of the worst.