A photograph of a street contains, in principle, a great deal of commercial information. The names of the businesses, what they sell, whether they look open, whether the frontage has been refitted recently. Extracting that automatically is one of the more obviously useful things a vision model can do with street imagery, and it is attempted constantly.
It also fails in ways that are easy to miss during evaluation and expensive in production, because the failure is not a blank output. It is a plausible business name attached to the wrong building.
The gap between reading text in an image and knowing what business occupies a premises is much wider than it looks, and most of the difficulty is not in the reading.
What a storefront read is actually trying to produce
The naive framing is optical character recognition: find the text, transcribe it, done. That is a solved-enough problem and it is not the task.
The task is to produce a structured claim about a location: at this position, there is a business, its name is this, its category is that, and here is how confident the system is in each. The text on the sign is evidence towards that claim. It is not the claim.
The chain of inferences between the two is long, and each link can break independently:
- There is a business here rather than a residence, an office, or a vacant unit.
- The text visible belongs to that business rather than to a neighbour, an advertisement, or a passing vehicle.
- The text is the business name rather than a product, a brand it stocks, a slogan, a franchise operator, or a phone number.
- The premises are at this position rather than the position of the camera or the centroid of the building.
- The business is still there.
A pipeline that treats the output of a text detector as a business record has silently asserted all five, and it has no mechanism to notice when any of them is false.
Why this is harder than detecting text
Occlusion is constant. Street furniture, parked vehicles, delivery vans, pedestrians, awnings, signage from adjacent premises, and street trees all sit between the camera and the shopfront. In busy retail streets, which is where the businesses are, occlusion is normal rather than exceptional. A partially occluded sign is worse than a fully occluded one, because a fully occluded sign produces nothing while a partial one produces a fragment that a language-capable model will happily complete into a real business name.
Viewing angle degrades everything. Imagery captured from the road sees shopfronts obliquely, often severely. Text on a plane viewed at a steep angle compresses, letters overlap, and rectification requires knowing the plane orientation, which has to be estimated from the same degraded image. Projecting signs perpendicular to the facade, common in older retail streets, are more legible from the road and harder to associate with the correct unit, because they hang above the boundary between two shops.
Glare and lighting are the third. Glazed shopfronts reflect the sky, the buildings opposite, and the sun. Internally illuminated signs bloom at night and disappear against bright sky by day. In many captures the interior of the shop is a black rectangle or a mirror.
Multiple signs per unit is the fourth and the most underestimated. A single small shop can display the operator's name, a franchise brand, several product brands supplied by wholesalers, a payment provider logo, a delivery platform sticker, opening hours, and a promotional banner. Choosing which is the business identity is a judgement, and in many markets the largest sign is a sponsor's brand rather than the business name. Beverage companies and telecoms operators fund shop signage in exchange for prominence, so the biggest text on a small grocery is frequently the name of a soft drink company.
The consequence is a category of error that looks nothing like a recognition failure. The text was read perfectly. The wrong text was chosen.
The dashcam problem
Much of the street imagery available at scale, particularly outside areas covered by dedicated survey fleets, comes from vehicle-mounted cameras: dashcams, delivery fleet cameras, and crowd-sourced captures from phones on windscreens.
That imagery is optimised for the wrong thing. A dashcam faces the direction of travel, because its purpose is to record the road ahead. Shopfronts are perpendicular to that direction, so they appear at the extreme edges of the frame, at the most severe angles, in the region with the worst lens distortion, and for the fewest frames.
Several consequences follow:
- The usable crop is small and marginal. A shopfront may occupy a narrow slice at the frame edge for two or three frames as the vehicle passes.
- Motion blur is worst at the edges, because angular velocity relative to the camera is highest there. A vehicle at urban speed produces significant blur on nearby facades.
- The near side of the street is close and steeply angled. The far side is more frontal but further away and frequently occluded by traffic.
- Coverage is biased by driving patterns. Streets vehicles drive along are covered; pedestrianised streets, arcades, markets, and courtyards, which hold a large share of small retail in many cities, are covered barely or not at all.
None of this makes dashcam imagery useless. It makes it a source whose geometry has to be modelled rather than ignored. Practical responses include selecting the frames where a facade is closest to frontal rather than where it is largest, aggregating a read across multiple frames of the same premises, weighting confidence by observation angle, and being explicit that the two sides of the street have different error characteristics.
Panoramic capture from a dedicated rig avoids the framing problem and does not avoid the occlusion, glare, or sign-selection problems.
Hallucination on ambiguous crops
The failure that causes the most damage is specific to models with strong language priors, which now includes most of the useful ones.
Given a crop containing a partially legible sign, a model with a good prior over business names will produce a complete, well-formed, plausible name. It will not produce a fragment. The prior fills the gap, and it fills it with something that exists somewhere, because that is what the prior encodes.
This is the same mechanism that makes such models useful on clean inputs, and on degraded inputs it produces output indistinguishable in form from a correct read. A blurred sign becomes a confident name. A crop containing a partly visible chain logo becomes that chain, even if the occupant is an independent business that inherited the signage frame. A crop of the shop next door becomes the neighbour's name attached to this position.
There is a second-order version that is worse. Given context, such as the city or the surrounding businesses, the prior sharpens and the hallucinations become more plausible rather than less frequent. A model told it is looking at a particular district will generate names that fit that district, so output quality appears to improve because the errors become harder to spot.
The mitigations that work are structural rather than prompt-level.
- Make abstention a first-class output with an explicit representation, and evaluate it. A pipeline with no way to say unreadable will never say it.
- Score legibility separately from identity. How much of the sign was visible, how sharp it was, and how frontal the view was can be computed independently of what the sign said, and that is a better gate than the model's own confidence.
- Require agreement across independent observations. If three passes of the same premises produce three different names, the correct output is unresolved, not the modal answer.
- Penalise fluency. A returned name that does not appear as a contiguous string in the raw character-level detection is a candidate hallucination, however plausible.
A confident wrong name is worse than an abstention by a wide margin, and the reason is downstream rather than local. An abstention leaves a gap a later process, a human, or a different source can fill. A wrong name fills the slot, blocks the gap, propagates into whatever consumes the dataset, and gets counted in coverage metrics as a success. Systems tend to be measured on how many records they populate, which is precisely the incentive that produces confident wrong names.
Return a reason, not just a tag
A read that says pharmacy is a claim a human cannot check without reopening the imagery. A read that says pharmacy, because a green cross sign was visible above the entrance and the window displayed prescription signage, is a claim a human can accept or reject in seconds.
Returning the evidence alongside the conclusion changes the economics of review, which is the constraint that determines whether a vision pipeline is usable at scale. Review time per record is the bottleneck, and evidence collapses it.
The evidence worth returning:
- The raw detected text, before any interpretation, exactly as characters were recovered.
- Which visual elements drove the category.
- The crop or a reference to it, with the frame identifier and timestamp.
- The observation geometry: distance, angle, and estimated occlusion.
- Competing hypotheses and their scores, where more than one candidate was plausible.
The last is undervalued. A read that says most likely a bakery, possibly a cafe, with both signals present, beats a forced choice, because it tells a reviewer where to look and tells an automated consumer that the categorisation is soft. An agent receiving a bare category has no basis for deciding whether to trust it. An agent receiving a category, a reason, and an observation geometry can apply its own threshold and ask for a different source when the geometry is poor.
Staleness is a property of the photograph
A vision read describes the moment the shutter opened. Nothing more.
Retail turnover is high, and highest in exactly the categories street imagery captures best: independent food, convenience, services, and small retail. A capture from a few years ago describes a set of businesses that has partly turned over, at a rate varying by category, location, and economic conditions in ways no single decay constant captures.
The capture date, not the processing date, is the date of the observation. Pipelines that stamp records with the date the model ran are recording when a computer looked at an old photograph, which is not information about the world.
Signage lags reality in both directions. A closed business leaves its sign up for months. A new business trades for weeks under the previous occupant's frontage while its own is fabricated. Signage is a lagging indicator of occupancy, and a read from a photograph is a lagging indicator of the signage.
Confidence should decay with age, and the decay should be visible in the record rather than applied silently. A consumer who knows a read is four years old can weigh it. A consumer given a pre-decayed score cannot tell a fresh uncertain read from an old confident one.
Privacy is not optional here
Street imagery contains people, vehicles, and the interiors of premises visible through glazing. Processing it for business identity does not exempt a pipeline from the obligations attaching to the rest of the frame.
Faces and vehicle registration plates are the obvious cases and are usually handled by blurring at ingestion. The obligation does not end there. Shopfronts are frequently residential premises at the same address, and a read that associates an identifiable person's home with a business, or that captures a customer through a window, creates a record that was not intended.
The less obvious risks are aggregate. A sequence of captures over time reveals patterns of activity at a premises, and combining a business read with vehicle presence produces inferences about individuals that neither source produces alone. Small businesses are often indistinguishable from the individuals who run them, so a claim about the business can be a claim about a person.
Reasonable practice includes blurring identifiable individuals and plates before any downstream processing rather than at publication, cropping tightly to the signage region so interiors are excluded from stored derivatives, retaining the minimum imagery necessary to support review, honouring removal requests through to derived records, and treating residential frontages differently from commercial ones by default.
Licence terms on the source imagery matter too, and they frequently restrict derived works or require attribution that has to survive into the output. That is easier to satisfy if provenance is carried per record from the start.
One signal among several
The correct position for a vision read in a data pipeline is as one input to a business record, never as the record itself.
Corroboration is available from sources that fail differently. Official registers are authoritative on legal identity and poor on location and trading name. Open map data carries points of interest contributed by people who were physically present, which is strong evidence of existence and weak evidence of currency. Address datasets constrain what can be at a position. Review platforms are current but skew towards categories that generate reviews.
Where a vision read agrees with an independent source, confidence should rise sharply, because the error modes do not overlap. A model hallucinating a plausible name and a register containing that same name at that same address is not a coincidence worth explaining away.
Where they disagree, the direction matters. A register entry with no visible signage suggests a closed or non-trading premises. Visible signage with no register entry suggests an informal or recently opened business, which in many markets is the majority case and exactly the population official sources miss. Neither should be resolved by silently preferring one source.
The read is most valuable precisely where other sources are weakest, in small, informal, unregistered commerce. That is also where it has nothing to be checked against, so it has to carry its own uncertainty honestly rather than borrowing credibility from the cases where corroboration was available.
The line worth holding
The output of a vision model looking at a shopfront is an observation about a photograph. It becomes a claim about the world through a chain of assumptions, and every one of them can be recorded, or can be buried.
A pipeline that records them produces a dataset where a wrong entry can be traced to the frame that caused it and the assumption that failed. A pipeline that buries them produces a dataset of confident business names, some proportion of which are the shop next door, with nothing to indicate which.
The second kind is cheaper to build, scores better on coverage, and is the one that gets built.