Somebody has shown you a demo. A camera feed, boxes drawn neatly around objects, a counter incrementing in the corner. It looked convincing, the price was plausible, and now you have to decide whether to put it on your own site, where the light is bad, the weather is real, and a delivery van parks across two bays every afternoon.
This is the decision most vision projects are won or lost on, and it is made before any code is written. The technology is mature. Detecting objects in a frame is a solved problem in the sense that the hard research was done and the tooling is widely available. What is not solved, and what decides whether your project works, is whether it holds up in the specific conditions of your specific site.
This guide covers what a demo does and does not tell you, the site conditions that determine the outcome, what to ask a vendor for before you commit money, how to agree on what good enough means, and the privacy questions worth settling at the start.
A Demo on Clean Footage Predicts Very Little
Every vendor demo works. That is what makes them nearly useless as evidence.
Public datasets and vendor reels are clean by construction: good light, clear sightlines, objects positioned the way the camera expects. The widely used research collections, COCO among them, were built to advance the field rather than to represent your loading dock at four in the afternoon. A model shown on that material performs excellently right up until conditions depart from it, which is the moment an operator needs it.
- Ask what footage the demo is running on. If the answer is a dataset, a reference site, or a controlled recording, the demo has told you the vendor can operate the tools. It has told you nothing about your site.
- Ask what it looks like in the worst hour of the day. Low sun down the length of a row, headlights blooming, standing water producing a second copy of every object. If the vendor has not asked you when your bad hour is, they have not thought about your site yet.
- Separate the model from the system around it. A detector that finds objects in a frame is one component. Whether the output is stable, auditable, and useful to an operator is a software engineering question, and it is where most of the work sits. That distinction runs through everything in our AI engineering practice.
The Site Conditions That Actually Decide the Outcome
Vision projects fail on physics far more often than on machine learning. These are the variables worth walking your site for before anybody quotes.
- Camera placement. Cameras installed for security were positioned to let a human recognize an incident afterwards, not to let software measure something continuously. A camera that cannot resolve the far end of a row will not resolve it at any resolution setting, and the correct answer for that position is to move the camera.
- Glare and low light. Direct sun into a lens washes the frame out for part of the day, and it does so seasonally, so a system commissioned in June meets a different problem in December. Night lighting brings its own difficulties: color shifts, hard shadows, and sensor noise that makes small objects disappear.
- Weather. Rain on the lens, standing water acting as a mirror, fog, snow on a housing. Outdoor deployments spend a real part of the schedule on conditions a demo never includes, and that is where the operator forms a judgment about whether the system works.
- Occlusion. Objects behind pillars, behind each other, or behind something temporarily parked in the way. Partial occlusion is not an edge case on an operating site, it is most Tuesdays. The question is whether your requirement can tolerate an object being invisible for a period, and what the system should report while it is.
- Change over time. Cameras get knocked. Lines get repainted. A row gets turned over to a different use. A system that has to be retrained every time the site changes will quietly stop being maintained. Ask how the design absorbs physical change.
- Network and power at the edge. The uplink at an operating facility is frequently a business line shared with everything else on site, and video is the one payload guaranteed to saturate it. Whether inference runs on site or in the cloud carries real operational consequences, including what happens during an outage.
Ask for a Survey Against Your Own Footage
The most useful thing you can buy before committing to a vision project is a short, paid feasibility study run against your recordings. This is not a formality. It is where the real constraints surface, and it is cheap relative to discovering them in month four. We treat a camera survey as a precondition rather than an option, which is exactly how the parking occupancy system in our work began: two camera positions on that site had been mounted for an entirely different purpose and could not resolve the far end of a row, and the honest recommendation was to move them rather than to promise a model would compensate.
- Give them your actual recordings. Pull samples deliberately across hours, weather, and seasons rather than uniformly. A week of footage from one clear month is not a sample, it is a best case.
- Expect a per-position verdict. The output should name which cameras work as mounted, which need moving, and which cannot be made to work at all. A survey that says everything is fine has not been performed.
- Expect the conditions to be enumerated. Which situations will degrade performance, roughly how often they occur on your site, and what the system is designed to do while they last.
- Ask what the study leaves you with. A findings document you own and could hand to any competent team is worth more than a proposal. That is the shape of the smallest engagement we offer, and there is more on it in our process.
- Ask about integration early. Whether your recorders speak a standard interface such as ONVIF or RTSP, and where the output needs to arrive, affects scope more than most people expect. There is a broader version of this question in scoping a connected product.
Define Good Enough in Operational Terms
Vendors quote accuracy figures. Operators experience something else, and the gap between those two is where disappointment lives. A single accuracy number hides which errors are happening and under what conditions, so two systems can report the same figure and behave completely differently on your site. The useful conversation is about the shape of the errors and what each one costs you.
- Decide which error is expensive. Missing something that is there and reporting something that is not are different failures with different consequences. Telling a driver a space is free when it is not costs you a complaint. Telling them a few seconds late costs nothing. Say which way you want the system to lean, in writing, before development starts.
- Specify behavior under uncertainty. What should the system report when it cannot tell? Holding the previous state, reporting unknown, and guessing are three different products. This is a design decision, not a model property.
- Agree how quickly it must settle. A system that changes its mind twice in five seconds can be technically correct and operationally unusable. Stability requirements belong in the specification alongside everything else.
- Ask for the full breakdown, not a headline. Results reported per condition and per class, on your footage, with the failures shown rather than summarized. A vendor confident in their system will show you where it struggles.
- Write the acceptance test as an operational scenario. Describe a day on your site and what the system must do throughout it. That is something both parties can check, and it is far harder to argue about afterwards than a percentage.
Settle the Privacy Questions Early
This is the part that gets deferred and then rewrites the architecture. It costs almost nothing to settle at the start.
- Fix the label schema first. What the system detects is the foundation of every privacy conversation that follows. A schema containing vehicles and nothing else is a straightforward assessment. Add plates or faces and you are in a different regulatory conversation, with different obligations depending on where you operate.
- Decide what is retained and for how long. Frames kept for retraining, frames kept for incident review, and frames not kept at all are three separate policies. Write them down, including who is allowed to access each.
- Consider whether identification is required at all. Many operational questions can be answered by counting and measuring without ever identifying anybody. If your requirement does not need identity, excluding it removes a large body of obligation and is the cheaper design.
- Bring counsel in before development, not after. The conversation is short when the answer is already a documented policy. It is long and expensive when it starts with a system that already collects more than it needs.
- Ask where the data goes. On-site inference that emits a few hundred bytes of state is a different proposition from streaming video off site for scoring, both operationally and in what you have to explain to your own stakeholders.
What a Realistic Program Looks Like
A vision project on a working site is mostly ordinary software engineering with a model somewhere in the middle. The model answers one narrow question about a frame. Everything an operator relies on, the rules, the stability, the history, the interface, is deterministic code above it, and keeping those layers apart is what lets the site change without the system being rebuilt.
Approached that way this is a tractable, well-understood project: a survey against your footage, an honest verdict on your cameras, an operational definition of success, and then a build against real conditions rather than clean ones. Skip the first step and every later decision is a guess.
If you are weighing a vendor now, the questions in what to ask a development partner apply here as much as anywhere. And whether your site is a good candidate at all is the kind of thing our AI engineering practice prefers to establish before a proposal rather than after one.