Why video object detection hits a latency wall
In a video object detection pipeline, the bottleneck is almost never the model itself. The decoder outputs frames at a fixed rate, while per-frame processing takes variable time: the scene gets more complex, there are more objects in frame—and time per frame increases.
The difference accumulates in the frame queue, and after a few minutes the system is analyzing what happened on site a minute ago.
Frame queue overflow is the first thing visible in metrics: the buffer hits its limit, and after that a deliberate policy is needed.
If old frames are dropped, detection runs on fresh frames, but the continuity of the object track breaks; if new frames are dropped, output latency grows, and the operator sees the past.
With no explicit rule, the library makes the decision, and usually it is not the rule the task needs.
Peak load finishes the picture: several camera streams start simultaneously, and the drop across several streams compounds rather than averages out.
Dropping frames under peak load is not just a drop in accuracy: some events remain unnoticed, and they have to be reviewed after the fact from recording—if it was even saved in sufficient quality.
Hence the requirements: latency must be predictable, and behavior under load must be defined in advance, not discovered during an incident.
We break down how this is covered: how to allocate resources between streams, where to deliberately cut resolution and frame rate, which metrics to watch so that lag does not accumulate.
What rebuilding the detection pipeline delivers
Rebuilding the stream processing pipeline is not replacing the model, but separating the work into independent stages: ingestion and decoding, model inference, tracking, archive recording.
Each stage has its own queue and its own resource, so a growing number of cameras no longer brings down overall throughput.
Distribution of load between nodes
Inference on TensorRT is moved into a separate pool from decoding and post-processing, and they scale independently. Overflow of one queue no longer stops the others: excess frames are dropped by priority instead of piling up in the buffer.
The share of frames lost to overflow goes to zero, and the bottleneck is visible in metrics rather than guessed.
Stable latency under load
Queue depth is bounded, and priority is given to the freshest frame: under peak, the system skips stale frames instead of accumulating them. Output latency stays in the same corridor when the number of streams doubles—the operator sees an event while it is still relevant.
Headroom in number of streams
Throughput is measured in advance on a test profile: you can see how many cameras one node holds and at what point a second one is needed.
Connecting a new stream does not require rewriting logic—only the handler pool is expanded, so headroom is planned rather than discovered at launch.
Eliminated class of failures
Previously, a crash of one process stopped the entire pipeline; now the stages are isolated, and restart takes seconds without archive loss.
A separate class of failures related to memory leaks in long sessions is also gone: the handler is recreated on a schedule, not after a stoppage.
Stream processing modes: batch, streaming, hybrid
Stream processing mode is chosen not by taste, but by what happens to the frame after detection: whether it goes to archive, waits for an operator decision, or immediately triggers an action.
This determines where the frame queue sits, how many inference accelerators are needed, and what latency arises between an event and the system’s response.
| Option | When it fits | Mode limitations |
|---|---|---|
| Batch archive processing | review of recordings for a past period, summaries, violation search over already captured material | an event is detected with a delay—reaction at the moment it happens is impossible |
| Real-time processing | zone monitoring and reaction to an event at the moment it appears | the frame queue must not grow: with insufficient compute resources, frames begin to be dropped |
| Hybrid scheme with an edge node | many cameras, weak or overloaded communication channel to the center | boundaries are set by the edge node itself—its resources and the complexity of updating models |
How to choose a mode for the task
Count not the number of cameras, but the latency and completeness requirements. If the result is needed from recorded material, batch mode gives more even accelerator utilization and allows models that cannot keep up with live stream pace.
When an event requires action in the same second, streaming mode is used and the frame drop policy is defined in advance.
Switching between modes
Modes are not mutually exclusive: the same pipeline serves the live stream and, on a schedule, picks up archive on the same models. Only the frame source and queue priority change—under overload, some frames go to batch review instead of being lost. Thus switching to a new mode does not require rewriting detection logic.
How implementation proceeds: from stream audit to handover
Implementation is split into short stages, and each ends with a verifiable artifact—a report, diagram, or configuration. We start not with model selection, but with analyzing what actually comes from the cameras and which events need to be recorded.
- Video stream and scenario audit. We look at codecs, resolution, night shooting, occlusions and blind spots, object behavior at frame boundaries. Here we also record the list of classes and the acceptable miss rate—these become the future acceptance criteria.
- Processing placement scheme. We decide what is computed on the camera, what on a local node, and what in the center: where latency is more critical, where the channel is cheaper, where the system must survive a communication outage.
- Pipeline setup. We assemble the decoding, detection, and tracking chain, set sensitivity thresholds separately for each scenario, and prioritize heavy streams.
- Calibration on archive recordings. We run reference clips, calculate accuracy and completeness by class, and adjust thresholds to agreed values.
- Acceptance on a real stream. A run on production cameras during working hours, including peak scenes and night.
- System handover and training. We provide configurations, placement diagram, and maintenance regulations, train operators, and support the first weeks of operation.
What counts as acceptance
Acceptance is not based on a demo clip, but on a 24-hour run on production cameras during working hours. We calculate the share of correct detections, the number of false alarms per camera per day, and the latency from event to notification—these numbers close the stage.
Mixed camera fleet
Cameras of different generations output streams differently, and some sources are weaker than others. For them, we lower the resolution of the analyzed frame or move processing to a local node—replacing the fleet is not required.
What we hand over: configurations, diagrams, regulations
We hand over the system not as “it works for us,” but as a reproducible loop: the on-duty shift brings it up on a clean node using the document, without messaging the authors. What is included in the delivery:
- Pipeline configurations: decoding, batch size, confidence and tracking thresholds—in a single versioned file; changes are applied by restart without rebuilding the image.
- Stream and node diagram: where video comes from, where inference is, where event and metadata storage is, and through what alarms go to adjacent systems.
- Access and role matrix: who changes thresholds, who restarts the service, who only reads the archive; separately—service accounts for cameras and queues.
- Operations regulations: startup and shutdown procedure, stream liveness checks, behavior when communication with a camera is lost—reconnection with increasing delay instead of a request storm.
- On-duty and incident handling instructions: typical failures, what to check first, what to escalate, and within what timeframe.
- Deployment scripts and a verification run on a reference recording—acceptance is reproducible, not taken on faith.
Documentation package for the system
Architecture description of nodes, a reference for each configuration parameter with permissible bounds, an update guide with a procedure for rolling back to the previous release, and a change log—so that six months later it is clear why the thresholds are set precisely this way.
Case: frame loss during peak hours—what changed
At a production site, video object detection ran in the same process as archive recording, and during peak hours the pipeline could no longer keep up with the stream.
Failures were discovered from discrepancies with recordings rather than from alerts, so measurements were made on already accumulated data for a week.
Initial picture: where frames were lost
The share of missed frames during peak intervals reached 9–12%, and the latency spread between frames differed by multiples: the detector either responded instantly or lagged.
False detections accounted for about a third of alarms, and after a process crash the service was brought up manually—recovery time after a failure was measured in tens of minutes and depended on the on-duty operator.
Measurements after implementation
The pipeline was split into independent stages—capture, inference, post-processing—with fixed-depth queues and dropping of stale frames instead of accumulation.
Inference was moved to the current TensorRT branch with input tensors normalized to a single format, and detector thresholds were calibrated on recordings from the site itself rather than on public datasets.
Repeated measurements during the same peak hours: frame misses—a fraction of a percent, latency spread decreased by about three times, and the number of false detections fell by two thirds.
Recovery after a failure became automatic—the stage restarts without on-duty involvement, and analytics and archive are not interrupted.
What to do if the model is already deployed but cannot handle the load?
If the model is already deployed and produces results, but cannot handle the load, the cause is more often not the weights but the loop around them: CPU preprocessing, synchronous calls, extra frame copies, lack of batching.
The first step is baseline measurements: we break down time into decoding, preprocessing, model inference, post-processing, and rendering; we look at queue occupancy and the share of dropped frames.
From these numbers it is clear what to reuse and what to rebuild.
We go through refinement rather than replacement: we preserve weights, labeling, exchange contracts, and result storage; we change the surrounding layer—preprocessing moves to the accelerator, an asynchronous pipeline appears with a bounded queue and eviction of stale frames, and batches are assembled by timeout.
We accelerate inference on TensorRT: we build an engine for the specific device and profile layers, rebuilding the most expensive parts of the network. If the constraint is latency rather than completeness, we deliberately skip some frames—the policy depends on the task.
Work order step by step: measurements and profile, then loop fixes—queues, batches, memory, then replacement of the runtime environment, and only then model lightening: quantization or distillation.
The sequence depends on access to the stream before encoding and on whether detection is needed on all cameras or only some.
We bring up the new loop alongside the old one and switch traffic camera by camera: comparison runs on live data, and rollback takes minutes.
Accuracy, false detections, and support: common doubts
We collect the questions that most often come up during project review before launch. We answer substantively and without promises of “perfect accuracy”—on a real site, system behavior is determined by noise in the frame, network quality, and the cameras already installed.
How is accuracy calculated and how many false detections will there be?
We do not show a single averaged number, but break the result down by class and shooting conditions: day, night, rain, backlight. In the report we record two metrics—the share of missed events and the number of false detections per shift.
We set the confidence threshold for your task: in some cases it is more important not to miss, in others not to raise an operator over trifles.
Will already installed cameras work?
We connect the existing fleet and check it before start: angle, height, actual stream from the camera.
If the model physically cannot provide the required image or the point is in a blind spot, we say so in advance and propose locations where replacement or repositioning will have an effect. We will not silently build a system on unsuitable data.
What happens if communication is lost or a node restarts?
The node processes the stream locally and continues detection without an external channel. Frames from the communication loss period go into a local buffer; after recovery it is processed—events are not lost.
On restart after power loss, the node comes up on its own, detection state is restored from the local journal, and manual intervention is not needed.
Can the model be fine-tuned on our data?
Yes, and usually this is a mandatory stage. The initial model is trained on open datasets, then on your recordings from the site. We label the disputed frames where the system is uncertain, so the fine-tuning cycle is short. A new version reaches the site only after validation on a holdout set.
How is quality checked after handover?
We provide not only the model but also a reproducible set of checks: test scenes, expected result, run procedure. You can run it yourself at any time and see where the solution has degraded. It also serves as a baseline when lighting changes or cameras are added at the site.
Let’s discuss the task: we’ll send an estimate and work plan
For the estimate to be realistic rather than “market average,” a minimum of context is needed. Below is the data for an estimate that is usually sent at the start: this is enough to calculate the scope of work and the order of stages without a week of clarifying rounds.
If some information is not yet known, write that—we will include assumptions and verify them at the first stage.
- Scenario and goal: what we detect and where a human makes the decision, and where automatic reaction is needed.
- Video streams: number of cameras, their type—fixed, PTZ, recorders—whether there is night shooting and low light.
- Where the archive lives and how the stream is delivered: local recorder, network storage, or cloud; by which standard video surveillance protocols.
- Reaction speed requirements: acceptable latency between event and signal, and what happens next—log, operator notification, call to an external system.
- Placement constraints: closed loop without internet access, customer servers or cloud, personal data storage requirements.
- Examples of recordings and labeling: whether there are already captured clips, which object classes are important, and what counts as a false detection.
In response, we send an estimate with the scope of work, a stage-by-stage plan—from data preparation and model training to validation at your site—and acceptance metrics: detection completeness, false detection rate, and signal latency. Let’s discuss the task—we’ll come back with an estimate and plan.
