Why sign language recognition AI fails on live video
In sign language recognition, the pipeline confidently passes a labeled recording — and falls apart on the very first live dialogue. The problem is not in the model itself: the training corpus is collected in close-ups, under even lighting, and with pre-cut fragments. Live video removes all these assumptions at once.
Hand overlap, shadow, going off the edge of the frame — the pose estimation model loses key points. The tracker jumps between left and right hand, coordinates jitter from frame to frame, and what goes down the pipeline is no longer a gesture but noise. An input error is not smoothed by the temporal model but multiplied by it.
The temporal part — a transformer with CTC decoding — is trained on an average tempo. In reality, service phrases flash by twice as fast, names are shown more slowly, and pauses do not coincide with the boundaries of semantic segments. Frame-to-symbol alignment drifts, and the output contains gaps and duplicate signs.
A separate source of loss is the desynchronization of video and timestamp. Frames are buffered and lost when the channel is overloaded, the camera clock drifts relative to the server, and lighting changes alter contrast between adjacent fragments. Features stop corresponding to the moment of capture.
On a test bench, each failure looks harmless; on a live stream, they add up to a collapse in accuracy. We analyze how this is addressed.
What changes after implementation: speed, accuracy, load
The key metric is the time from the completion of a gesture to the appearance of text on the screen. It consists of frame processing, passing through the skeletal hand model, and accumulating a window of frames on the temporal model.
In live shooting with a regular camera, the delay can be kept within fractions of a second, and the viewer reads the stream as simultaneous interpretation, not as a transcript afterwards.
The interface response is deliberately separated from recognition itself: text is output word by word, the current fragment is highlighted, and the recognized sentence remains on screen while the next one is being formed.
This removes the feeling of a choppy stream and allows making decisions based on an incomplete phrase without waiting for its end.
Proportion of confident recognitions and model refusals
The second indicator is the proportion of confident recognitions on a live stream, that is, gestures where the model produces a result without a conflict of hypotheses.
Refusals here are of different classes: uncertainty on a single frame, discrepancy between the gesture and the accumulated phrase context, and loss of hand tracking when leaving the frame.
Instead of a random hypothesis, the system marks the fragment as unread and asks the operator to repeat — the proportion of unnoticed errors falls, and disputed places are visible directly in the interface.
We test robustness to interference by changing conditions: a different operator, a different gesture tempo, background, lighting, shooting at an angle.
Some discrepancies are removed by fine-tuning on a short recording of a specific signer — the proportion of confident recognitions returns to the level of reference recordings.
The load on the capture device is manageable: with a weak device, computation is moved to the server, and the camera only sends frames, or the model is compressed to work on-site.
The result of implementation is three measurable effects: delay until text appears, proportion of confident recognitions, and the class of refusals, which the system shows honestly instead of guessing.
Architecture options: streaming model, hybrid, offline analysis
The scheme is chosen not by the quality of the model, but by where the video appears and when the result is needed. Hence three basic schemes — streaming recognition, offline video analysis, and a hybrid with server-side refinement.
| Option | When it fits | Limitations |
|---|---|---|
| Post-hoc recording analysis | Lectures, educational assignments, archive: transcription needed after recording | No real-time translation, sensitivity to recording quality |
| On-device streaming | Live dialogue, scene without stable connection, local frame processing | Device resources are limited, some gestures require refinement |
| Hybrid with server-side refinement | Real-time translation with high accuracy | More complex to operate: synchronization and delay control |
| Ready-made keypoint libraries + custom model | Quick start and control over gesture vocabulary | The library provides points but not meaning — quality is maintained by the custom model |
| Rented cloud service | First runs, rare sessions, no own infrastructure | Dependence on someone else's update pace and data restrictions |
When recording analysis is sufficient
If a gesture does not need to be translated at the moment of display, the model sees it in full and can double-check disputed areas. This is suitable for transcribing lectures, analyzing educational assignments, and processing archives. The delay from the end of the recording to the result does not interfere here.
When on-device streaming is needed
For live dialogue, frames are processed locally, and the result appears immediately after the gesture — local frame processing also removes the issue of transmitting video outside.
The limitation is device resources and the vocabulary that the model parses confidently; rare and continuous gestures are more reliably refined on the server, which is what the hybrid scheme provides.
How implementation proceeds: from video audit to model handover
Implementation proceeds in iterations: first we understand which shooting scenarios are realistic, then we fix the gesture vocabulary — and only after that we train the model. This order prevents training the recognizer on studio video that does not resemble the working stream.
Labeling and gesture vocabulary
- Video material audit — we review recordings from target cameras: lighting, angle, background, gesture tempo, overlaps. We record which conditions the model must handle and which remain outside the scope of application.
- Vocabulary coordination — we consolidate the list of gestures with the client, discard visually indistinguishable pairs, agree on a neutral class and gesture boundaries.
- Gesture labeling — we label recordings by boundaries and frame by frame, disputed fragments are checked by a second labeler; discrepancies are resolved before training, not after.
Acceptance on a live stream
- Training and confidence threshold selection — we train the temporal model on labeled sequences and select the activation threshold so that misses and false positives are balanced for the scenario.
- Live stream testing — we run it on a real camera under target lighting: we look at output delay, behavior during rapid transitions, and when there are multiple people in the frame.
- Interface integration — we connect the recognizer to your application, deliver the result in an understandable format, and keep manual input as a fallback.
- Solution acceptance and handover — we agree on metrics, record them on a control sample, hand over model weights, labeling, training procedure, and instructions, and train your team to fine-tune the vocabulary.
What we hand over: models, schemes, regulations
We hand over not a model on disk, but a kit by which your team operates and develops sign language recognition without us. Each artifact is deployed in a clean environment and reproduces the metrics from the report.
- Model weights and configuration: training checkpoints, class confidence thresholds, temporal smoothing of predictions, "gesture not recognized" refusal conditions.
- Pipeline scheme: video stream intake, hand and pose extraction, temporal sequence assembly, decoding into text and output to your interface.
- Gesture vocabulary: classes with reference examples, start and end gesture labeling, rules for adding new classes.
- Video access policies: retention periods, roles, anonymization or refusal to store frames, access log to the stream.
- Retraining instructions: preparation of labeling, launching training, reproducibility of the run, rollout of a new version and rollback to the previous one.
- Quality report: metrics on a held-out sample, error analysis by class, validation on recordings from real shooting conditions.
Documentation and gesture vocabulary
We describe a gesture so that a labeler can add new ones without our help: where the movement begins and which execution variants are considered one class. The vocabulary lies next to the weights.
Regulations for fine-tuning and quality control
We record who launches fine-tuning, which metrics are acceptable, and at what quality drop the model goes for a repeat run. Rollback to the previous version and analysis of disputed recognitions are described separately.
Case study: live sign language interpretation in a clinic — metrics before and after
A clinic with a constant flow of sign language patients conducted appointments as follows: the request was registered, they waited for a free interpreter, the conversation was recorded, and transcription was done afterwards.
The recognition system was built on MediaPipe on top of a live video stream from the office — with the expectation that the hint appears at the moment of the gesture, not the next day.
Measurements before implementation
At the start, baseline metrics were taken. The proportion of requests closed without an interpreter was zero: any conversation required a specialist.
Transcribing a 40-minute appointment from a recording took about three hours, and up to a quarter of semantic blocks were lost in the manual protocol — about deadlines, follow-up visits, preparation for examination. The patient received a significant part of the information in writing and after the appointment.
What changed on the live stream
The translation delay on the live stream is about one second: the gesture is recognized frame by frame, the text on screen updates as the phrase progresses.
The proportion of confident recognitions where the model gives the answer itself, without human involvement, is 78%, the rest goes to an interpreter with a low-confidence mark.
Due to this, 6 out of 10 requests pass without an interpreter, and a specialist is involved only for complex formulations.
“Previously, an appointment with an interpreter had to be planned several days in advance and adjusted to their schedule,” says the head of the department.
“Now the hint goes along with the conversation, and we call an interpreter for clarifications on a case-by-case basis.” The client's feedback became part of our acceptance: metrics before and after were verified against the request log, not by impressions from a demonstration.
What to do if recognition already works but gives failures?
Failures almost never live in the model itself — first we check the input. Cropped hands at the frame edge, lost key points during fast movement, different scale and angle of the hand between recordings cause prediction jumps that are easy to attribute to a network error.
Therefore, the first step of the pipeline audit is to run reference fragments and see at which layer the result diverges.
Next is the gesture vocabulary. Classes labeled by different people diverge in boundaries and duration: one gesture is recorded as two, similar gestures are merged into one.
Labeling errors are almost invisible in overall accuracy but visible in the confusion matrix — a pair of classes is consistently confused, and the decoder produces a confident but incorrect result.
A separate cause is shifted confidence thresholds. After changing the camera, lighting, or background, the distribution of scores shifts, and a hard-set threshold begins to either miss gestures or trigger on noise.
We re-select the threshold based on the error curve on a labeled set, not on a single averaged indicator.
The verification order goes from input to output: video cutting and synchronization, hand detection, features, model, decoder, post-processing.
We cover the detection layer with ready-made libraries like MediaPipe to avoid reinventing the solved, and we focus mainly on features, vocabulary, and smoothing logic.
The refinement of the solution is integrated into an already running service without stopping it: the new version is launched in shadow mode next to the current one, comparison is done on live traffic, migration is done for fractions of users, and rollback is done with a single switch.
Thus, editing the vocabulary or thresholds does not bring down a product that is already bringing benefits.
Common engineer questions: accuracy, data, deployment
Engineers ask the same questions at the start: accuracy on rare gestures, data volume, video privacy, and autonomy. Below are short answers for each.
How accurately are rare and compound gestures recognized?
Rare classes underperform due to sample imbalance: we balance examples and add distortions by key points. We assemble a compound gesture using CTC decoding over the sequence, and in the report we calculate accuracy on rare classes, not on average across the vocabulary.
How many recordings are needed for training?
For dozens of gestures — hundreds of repetitions per class, and variation is more important: angles, lighting, tempo, different people. Plus recordings without a gesture, so the model confidently distinguishes "no sign" from a weak activation.
Where do video recordings go?
Video does not leave your perimeter. If necessary, only the coordinates of key points are transmitted outside — it is impossible to reconstruct a face from them, and the retention period and access are set by your policy.
Does the system work without a network?
Yes: point extraction and sequence analysis are done on-site, the channel is only needed for model updates. If communication is interrupted, recognition continues, and the result is sent later.
How to fine-tune on our side?
You label new gestures in the same interface, then fine-tuning on the combined sample. We check that old classes have not degraded, the model version is fixed, and rollback to the previous one remains possible.
Let's discuss the task: we'll assemble a scheme and come back with a plan
To ensure the project estimate is based on data, not assumptions, the materials you already have are enough to start.
- Video examples — 10-20 short recordings with different people, angles, lighting, and backgrounds: they show the real data spread, not an ideal demonstration frame.
- Gesture vocabulary — what the system should distinguish, where similar-meaning variants are acceptable, and where an error is critical.
- Usage scenarios — where recognition works and who views the result: an employee on shift, a teacher, a visitor at a service window.
- Requirements for response and accuracy — what delay still does not interfere with dialogue and what proportion of errors the scenario can tolerate without losing meaning.
- Data restrictions — whether video can be stored, whether there is a ban on filming people and moving frames outside your perimeter.
- Current process — how the task is solved now, how much time it takes, and at which step it most often breaks.
On the call, we clarify the vocabulary size, the labeling method, and who on your side confirms quality. In response, we send the pipeline scheme: how frames enter the recognition model, where false positive filtering takes place, and how the result is output to the user.
To the scheme we attach the work stages, the composition of artifacts, and acceptance agreements — which metrics we consider achieved and on which dataset we verify.
Describe the task — we will come back with an estimate and an implementation plan.
