Computer vision development: where the pilot breaks
A recognition pilot rarely fails on the model itself — it fails in the transition to production. On a prepared dataset, the metrics look convincing, but on a live stream the same algorithm produces detections that the operator stops accepting. Then refinement begins without a readiness criterion.
The first reason is uneven lighting and scene geometry. A model is trained on frames captured under convenient conditions, while on the shop floor lighting changes by shift, and overexposure and glare appear on metal and glossy packaging.
If computer vision development does not begin with an assessment of the camera, lighting, and viewpoint stability, accuracy drops even with unchanged code.
The second is a rare defect class. There are few defects, it accounts for fractions of a percent of the dataset: average accuracy across all classes remains high, but the system does not catch the specific rare class.
It is masked by the overall metric, so a separate measurement for each class and a plan for collecting additional examples are needed, not just an increase in the volume of labeling.
The third is unstable labeling. While the defect criterion is held in the operator's head, the same frame is labeled differently, and the model learns from contradictions.
This is treated with a regulation: class descriptions, review of disputed frames, rechecking part of the dataset, and inter-annotator agreement control.
The fourth is the absence of acceptance testing on a live stream. If it is not fixed in advance what share of misses and false positives we consider acceptable and how the system behaves on failure, there is nothing with which to accept the pilot. We explain how this is closed at each stage.
Implementation effect: quality control and reducing operator workload
We calculate metrics before launch, on a pilot sample of frames from the line: the share of missed defects, the number of false positives per shift, the time to review a disputed frame, and how many decisions the operator makes manually.
These are four values that show whether the system works — not the general "model accuracy" in a report.
After commissioning, the share of missed defects on controlled defect classes falls below one percent: the model inspects 100% of products, not a sample of every tenth one.
We suppress false positives with a confidence threshold and fine-tuning on disputed frames — a few remain per shift instead of hundreds, and the line does not stop for rechecks.
Computations run on NVIDIA accelerators: a frame is processed faster than the conveyor outputs the next one, so the line pace does not have to be reduced.
Reducing operator workload is a consequence: the system makes most decisions, while a person reviews only disputed frames and confirms the shift.
The time to review a disputed frame drops from minutes to seconds: the frame, the highlighted defect area, and the version of the model that produced it are right there.
Quality stability is provided not by a peak but by variance: an operator reacts differently at the end of a shift than at the beginning, while the model reacts the same, so the discrepancy between shifts and sites decreases.
We keep a separate log: which defects are missed, and on which materials the share of disputed frames grows. This is a ready-made fine-tuning plan and a clear boundary — what the system controls confidently and where the decision remains with a human.
Ready-made models, fine-tuning, or training from scratch
Three paths lead to a working model, and the choice of approach is determined not by fashion but by data: how many labeled examples you have on hand, how homogeneous the scene is, and what share of errors the customer considers acceptable.
A ready-made model solves the task in a matter of days — but only if your classes and shooting conditions match those on which it was trained.
| Approach | When it fits | Limitations |
|---|---|---|
| Ready-made model | Classes and scenes match the training set, standard accuracy is sufficient | Does not handle a non-standard viewpoint, a rare class, or a custom modality — validation on your frames is needed |
| Model fine-tuning (transfer learning) | Your own classes, but data of the same type: same camera, lighting, object scale | Requires labeling of thousands of frames; with a strong change in shooting conditions, features "freeze" and quality does not improve |
| Training from scratch | Unique modality: thermal imager, defects of a specific production facility, non-standard sensor | Long cycle, large volume of labeling, architecture and validation are designed from scratch |
We do not choose an approach by eye. First we run a ready-made model on several hundred of the customer's frames and see exactly where it makes mistakes — on a rare class, on overexposure, on night shooting.
If it handles typical frames confidently and falls apart only on your specifics, that is transfer learning. But if similar data does not exist in nature, we say directly: a ready-made model will not work, and architecture and labeling are needed from scratch.
Project path: problem statement, labeling, training, acceptance
We fix the acceptance criterion before the first line of code — otherwise acceptance turns into a debate about tastes. Each step has a checkpoint: until it is passed, the next one does not begin, and this is noticeably cheaper than retraining a model on poorly labeled data.
- Problem statement. We describe what counts as an event in a frame, under what conditions the camera operates, and what share of misses and false positives we consider acceptable. The outcome of the stage is an acceptance criterion in measurable values.
- Data labeling. We take real recordings of the site, including difficult frames: night, rain, occlusions, glare. Labeling follows a written regulation, disputed cases are reviewed by an arbitrator, and one fifth of the frames are checked by a second labeler.
- Model training. We fine-tune a base model and keep a separate holdout set that does not participate in tuning. We look not at average accuracy but at rare classes — usually it is they that break production operation.
- Validation on the stream. We run the model in shadow mode alongside the working process and compare its decisions with the facts. Here discrepancies between laboratory metrics and reality become visible: lighting changes, camera shift, a new type of object.
- Thresholds and robustness. We select the triggering threshold according to the cost of error and describe behavior when a frame is lost or the camera is unavailable — the system must degrade predictably.
- Acceptance and handover. A run on a control segment not used in either training or tuning, and a check against the acceptance criterion from the first step. We hand over the model, configuration, labeling regulation, and the procedure for rolling back to the previous version.
Delivery package: model, labeling, documentation, regulations
The delivery package is assembled so that the customer's team can train, validate, and replace the model without us.
Therefore it includes not only weights but also all layers on which reproducibility of the result depends: data and its labeling rules, training configurations, rights to work with video streams, and operating regulations.
- Trained model and checkpoints — working weights in the format accepted by your execution environment, plus intermediate snapshots by training stages: rolling back to the previous version takes minutes, not a new iteration of data collection.
- Datasets and labeling map — source and labeled datasets, class descriptions, rules for borderline and disputed objects. The labeling map removes the main source of discrepancies when new people join annotation.
- Training reproducibility — configurations, pinned dependency versions, shuffling order, and a quality control protocol. A repeated run produces comparable accuracy, and a discrepancy is explained rather than attributed to "magic".
- Integration schemes — how the model is embedded in the video stream, where failure points are, what latencies are expected at each frame processing stage.
- Video access policies — who sees the stream and archive and on what basis, retention periods, anonymization, logging of access to recordings. This also removes security service concerns when handing over the system.
- Operating regulation — what to do when accuracy drops, how a new version is released, who decides on retraining, how incidents and degradation during a change of lighting or season are recorded.
- Documentation package — instructions for operators and on-duty staff, description of quality metrics, acceptance and shift handover procedure, glossary of domain terms.
Quality control case: computer vision system on the line
Before implementation
On a polymer packaging filling line, control was selective: a QC operator took every twentieth container and inspected it visually. The defect miss rate reached 7% — microcracks and incomplete seams went to palletizing and came back as batch claims. False alarms accounted for 15%: the line was stopped for rechecking, and the shift pace dropped.
Model iterations and changes to the shooting scheme
The first version, on four thousand frames, reduced misses to 4%, but remained blind to rare classes — fewer than a hundred examples in the entire set.
We changed not the model but the shooting setup: side lighting at an angle to the neck, synchronization of cameras with a position sensor, filtering out blurred frames, and labeling of disputed cases in CVAT with double checking by an engineer.
The base grew to 16 thousand frames, and rare classes were expanded to three thousand examples.
The second iteration produced a 1.2% miss rate and 6% false positives; the third — 0.9% and 1.8%: we added confirmation across three consecutive frames so that a single glare on a label would not be classified as a defect.
Result
Defective containers are diverted by a signal without stopping the conveyor, and the operator deals only with disputed cases — fewer than two percent.
Weights and configuration are versioned, rolling back to the previous version takes minutes, and new defect types were added to labeling after launch: retraining took place without changes to the line's optics or mechanics.
What to do if cameras are already installed and there is a recording archive
You can start without replacing the fleet — the only question is whether the current cameras and recordings are sufficient for the task.
Inventory takes several days: we check viewpoints and mounting stability, the actual depth and completeness of the archive, lighting at the required hours, and spare network bandwidth.
Usually some cameras provide a usable stream, while others have to be excluded from the loop — their frames will only add noise to training.
Then we choose one of three starting scenarios: they differ in launch time and the amount of data needed before the first working model.
| Start option | When it fits | Limitations |
|---|---|---|
| Training on the archive | Recordings of the required scenes exist for several months, camera viewpoints have not changed | Quality depends on compression and night shooting; rare object classes will have to be supplemented through refinement |
| Additional equipment for shooting points | There is no archive or the viewpoints do not cover the area of interest | Longer launch: installation, lighting, calibration |
| Hybrid scheme | Some tasks are covered by current cameras, some are not | More complex setup and synchronization of events between loops |
When training on the archive, we label the sample in a stratified way: day and night, weekdays and weekends, season.
We take the holdout part from a different recording period — otherwise the metric on its own sample will be inflated and the model will underperform in the real loop. We perform preprocessing on open libraries, for example OpenCV, so as not to depend on closed licenses.
The hybrid scheme most often turns out to be the working one: existing cameras, server, and channels are reused, while additional equipment is deployed in a targeted way — lighting or an additional viewpoint where the model falls short on accuracy.
Such a start does not stop current processes, and the first measurable results appear on already installed equipment.
Technical concerns: lighting, labeling, false positives
Every recognition project runs into four things: lighting conditions, labeling quality, the triggering threshold, and behavior during failures.
These are discussed before launch because they determine how much manual checking remains for the operator and how quickly the system pays back the implementation.
Below are the questions most often asked by technical managers before evaluating a project.
What to do if lighting changes during a shift?
We do not rely on a single camera and a single light source. The scene is checked in the darkest and most overexposed conditions: backlight, glare from a wet floor, part of a shift.
If the brightness range is large, we add illumination in the invisible range and fix the exposure — the model sees a stable image, not a fluctuating one.
How do you know the labeling is high-quality?
We perform labeling according to a regulation: disputed frames go for a second review, and the share of discrepancies between labelers serves as a quality metric.
We label part of the data automatically and recheck it selectively — this is a faster way to build volume for rare classes, which usually fail on real scenes.
How do you reduce false positives?
We select the confidence threshold according to the cost of error: a miss and a false alarm have different costs. We do not discard disputed frames but send them for fine-tuning — the share of false alarms falls as data accumulates.
What happens when the model degrades or the camera fails?
We monitor data drift: if the frame distribution diverges from the training one, model degradation proceeds quietly, so metrics are calculated both on the control set and on the live stream.
If a camera is lost, the system switches to a safe mode — it records the event, passes it to the operator, and does not produce random decisions.
Let's start with your task: what to send for evaluation
Evaluating a computer vision project rests not on the algorithm but on the data and shooting conditions. Therefore we collect the input data for evaluation in the same way — according to the list below.
If you send this, we will come back not with general words but with a solution scheme and a plan of stages.
- Video or a frame archive from the site: different hours, lighting, weather, viewpoints.
- A list of objects and events: what needs to be detected, what we consider a miss, and what we consider a false positive.
- Shooting conditions: camera type, mounting height, field of view, whether there is pan and zoom.
- What should happen with an event: recording, a signal to the operator, a command to an adjacent system.
- Where processing is planned: on-site or on your servers, what access to the video stream is available.
- Constraints: data storage requirements, privacy, areas where shooting is prohibited.
- Acceptance criterion: by what metric you will consider the system working.
- Timelines and phasing: whether there is a hard launch date and what should work first.
If some of the data is missing, that's not a problem — we clarify these answers during the task discussion. Then we fix the solution scheme, the stage plan, and the scope of work for each.
