What breaks when recognizing behavior on a live stream from cameras
Behavior recognition on a live stream breaks not in demo mode but on a live shift. An event makes sense only if someone confirmed and handled it: when false positives come in batches, the operator stops responding to alarms, and a real incident passes by.
Night control zones are first failure. Infrared illumination produces noise on sensor, headlights blow out exposure, shadows from fencing read as silhouette.
A person disappears from frame for a few seconds, while "person in restricted zone" rule triggers on empty space.
Dense flow breaks object tracking. In queue at entrance people overlap, tracking line breaks and re-binds to neighbor — sharp movement is attributed to random person. Independent measurements of model accuracy in difficult conditions, e.g.
in NIST programs, show that solutions of one class differ in error rate by multiples; this gap determines how many alarms go to processing.
Then human factor. After twenty-thirty minutes of monotonous observation, operator attention drops, and if archive search is manual, reviewing one episode is hours of footage. Load on security grows while actual response falls.
All these failure points are closed by data preparation, rules, and thresholds, not by replacing one model with another — below we discuss how this is done.
What implementation gives: response time and false positive rate
First effect — time to notification. Frames are processed as stream, without accumulating segment, so alarm event reaches operator in seconds, not during shift review. Difference fundamental: in one case operator has time to intervene, in other only records fact.
Second — false positive rate. Threshold selected on recordings from site itself, not abstract sample: camera angles, lighting, workwear, typical personnel movements. On top of detector, context classifier operates — it filters shape matches where scene is actually normal.
The fewer false alarms, the higher shift trust and less often system is disabled "so it doesn't interfere."
Third — coverage of control zones. One incident caught in multiple cameras is merged into one event by person's trajectory, not turned into stream of duplicates. Therefore we add zones and cameras without expanding staff: operator gets more coverage, not more screens. We introduce new zone by setting area of interest, without retraining model.
Fourth — unloading duty shift. Manual viewing of recordings replaced by selective: they look by filters and metadata, not sequentially. Shift focuses on response to confirmed events, attention remains even by end of day.
Implementation effect measured over control period — time to notification, false positive rate, covered zones, and number of responses per shift.
Analysis modes: live stream, archive, and hybrid scheme
Analysis mode sets not only reaction speed but also requirements for communication channel, compute resources, and recording quality.
Live stream processes frames as they arrive and delivers event while it can still be prevented; archive analysis runs accumulated recordings in batch and gives a fuller picture due to repeated pass over difficult scenes. Hybrid scheme divides work: lightweight rules computed on live stream, heavy models on archive.
The fork is one: need reaction at event moment or enough post-factum review. For live stream we calculate latency budget and channel load, for archive — storage depth and batch processing speed.
Mode limitations appear at boundaries: online loses accuracy in low light, archive lags, hybrid requires event reconciliation between contours.
| Mode | When appropriate | Limitations |
|---|---|---|
| Rules and zones | Formalizable events: entry into restricted zone, line crossing | Does not understand context, triggers on similar allowed actions |
| Live stream | Reaction needed in seconds: fight, fall, intrusion | Sensitive to channel and lighting, some scenes have to be simplified |
| Archive analysis | Incident review, reporting, search through recordings for period | Reaction after the fact, result appears some time after event |
| Hybrid scheme | Critical zones on live stream, rest in batch | Two contours: more complex priority setup and event reconciliation |
In practice mode is chosen not for entire site, but for each zone separately — below we discuss how this affects markup, thresholds, and system support.
Implementation procedure: from zone survey to handover
Each stage leaves an artifact that next stage relies on. Skip survey — zone scheme will be put together from memory, and discrepancy will surface at acceptance, when rework affects both cameras and scenario logic.
- Site survey. We walk perimeter and indoor areas, record actual viewing angles, blind spots, lighting changes day and night. We immediately exclude events that cannot be reliably distinguished on current cameras.
- Zone scheme. For each camera assign control zones and rules: entry into restricted area, crowd gathering, left object. Agree scheme before setup — it also goes into shift instructions.
- Scenario setup. Select sensitivity by zones, removing false positives from shadows, glare, and glass, but not losing real events. Fix thresholds in configuration, not "by eye."
- Testing on recordings. Run scenarios on archive: shifts with known incidents and quiet ones. Calculate share of detected events and number of false alarms, adjust thresholds based on result.
- System acceptance. Customer confirms operation on live stream by checklist: response to control events, time to alarm in interface, correctness of notifications to operator.
- Shift training. Discuss how to read alarm, what to do on confirmation, and how to mark false positives — these labels go into retraining.
- Handover. Provide zone scheme, configurations, response regulations, and access; fix support procedure and rollback point to previous settings.
What stays with you: configurations, schemes, regulations
Documents are not an appendix to system but part of it: by them solution lives after delivery. Below — what exactly stays with you so configuration can be edited on your own, without waiting for us for every change.
- Working configuration of scenarios: zones of interest, sensitivity thresholds, activity schedule by hours and days, with explanation of why exactly this value is set. Values are edited without us — only logic changes, not code.
- Map of zones and blind areas: what actually enters frame, where there are overlaps by columns or racks, in which areas recognition works unstably. It is immediately clear where to add a camera to close hole.
- Access rules for recordings: who views stream in real time, who is entitled to export a fragment, retention period, access log. Roles are separated so viewing remains controlled.
- Response regulations: which event is addressed to whom and within what time operator must respond, what counts as false positive and how to close it in journal.
- Shift instructions: one-page cards — how to confirm or reject event, how to mark review, where to escalate abnormal case.
- Deployment scheme and failure behavior: restart procedure, what degrades first, how to roll back to previous configuration version without losing history.
- Solution handover procedure: accesses and roles, shift training, period during which we remain in contact on operation issues.
Package is handed over not as archive "for memory" but with on-site review: we go through each item with shift and technical service so instructions are tested on live events, not read for first time during incident.
Case: warehouse and night zones — how false positives were removed
Distribution warehouse: night shipping zones, duty shift of two operators, no video stream analytics. Alarms were raised by motion sensor — 300–400 events came to console per night, of which units were confirmed.
Operators stopped opening pop-up windows, and appearance of person in pallet zone was lost among shadow from loader and glare from headlights.
What changed
Detection was divided into two layers: coarse motion search in night zone and action classification by short fragment — person at rack, loader, shadow, animal.
Thresholds configured separately for each zone and lighting mode, and model retrained on recordings from this same warehouse: infrared illumination, rain, sudden activation of floodlights.
Model inference runs on edge node next to cameras, on NVIDIA, so frames do not go to center over communication channel.
What became
Measurements were taken on labeled sample of night recordings over same period before and after, on same zones and shifts.
- Confirmed triggers — from 8% to 71%: duty shift reviews a short list instead of a stream.
- Manual checks per night — from 350 to 40 events.
- Time to notification — from 40 seconds to 6: signal goes out while intruder is still in frame.
- Duty shift — one operator instead of two, second reassigned to patrols.
Zone rules and thresholds are in configuration, not in model code: returning previous mode or transferring settings to second warehouse can be done in minutes, without retraining.
How rule-based detection differs from training on markup
Rule-based detection is zone geometry and formalized conditions: object crossed line, entered restricted area, lingered longer than threshold, went against permitted direction.
Logic is deterministic: one episode gives one verdict, thresholds set manually, trigger explained in words. Hence model predictability: behavior is tested by test recordings and does not change from frame to frame.
Approach boundary — scenes where feature cannot be reduced to coordinates: swing, fall, waving arms, left object, fight. Rule "hand above shoulder" will not distinguish threat from greeting, and here markup is needed.
Training on markup begins with preparing examples: short fragments with behavior marked by intervals and a separate set of similar but allowed situations.
Model works with motion and pose features and transfers to new angles, but quality depends on completeness of markup: missed negatives and class imbalance cause growth in false alarms.
On site, combination of approaches works. Rules hold context — zone, time, direction, duration — and enable model only inside zone and after rule trigger. Model processes not entire stream but short interval, so load on compute and false positive rate fall.
Implementation order: rules close simple and explainable scenarios, then alarms confirmed by operator become examples for markup, and model is retrained on them. Cycle closes — operation itself grows example set, and rules and model work as one control contour.
Technical doubts before launch: what is asked most often
Technical questions before start concern not model itself but its integration with what already works on site: cameras, lighting, archive, access rights. Below — answers to what is asked most often.
Will already installed cameras work?
In most cases yes: stream goes over open standard, recorder does not transcode it on the fly. Before start, we take test fragments from typical points and check angle, detail, and overlaps — unsuitable camera is identified during measurement, not after launch.
What happens when lighting changes?
Light change is main source of false positives: when lights turn on or headlights appear in frame, picture changes more than behavior itself.
Dataset includes recordings from each point at different times of day and in bad weather, and trigger thresholds are configured separately by zones.
How long to store fragments?
We store not entire archive but short fragments around event plus metadata: time, zone, trigger type. Full recording remains in your system — we do not interfere with it, retention period is set by your policy.
Who gets access to recordings?
Rights are separated by roles: operator sees event and fragment but does not export array, security service gets extended access. Views and exports are logged.
How does retraining work?
Erroneous triggers and misses go to markup queue and are added to dataset. Retraining is done with validation on holdout set, and previous model version remains available for rollback.
Let's discuss the task and come back with estimate and plan
Accuracy of estimate and implementation plan depends not on number of cameras but on how completely scenes are described for which system will make decisions. Below — what to send for task review: drafts, exports, and screenshots are fine, no need to organize materials. Even a partial set already removes half the questions.
- List of zones with purpose: perimeter, checkout, warehouse, service passages, parking — separately for each.
- Camera placement scheme: angles, overlaps, coverage areas — it shows where scene is readable and where re-aiming is needed.
- Violation scenarios: what we consider an event — fight, fall, left object, passage into restricted zone. Examples of normal are just as important: what system should not highlight.
- Priorities by zones: which scenes are needed from day one, which can be enabled in second phase.
- Shift composition and response regulations: who watches notifications, during what hours and what they do afterward — this determines trigger threshold.
- Recordings for several typical days, including night and peak load — we will check recognition on them before work starts.
After review, we will come back with work estimate and implementation plan: order of zones, equipment and modifications composition, acceptance points for each scene. Describe task however convenient — we will clarify missing details ourselves.
