Live Captions: real-time subtitle delivery with latency <2 s

Live Captions: why real-time captions lag behind

Real-time captions are a pipeline: audio capture, slicing the stream into short windows, streaming speech recognition, text stabilization, and output.

Caption latency is the sum of each step's time, and if you optimize only the recognition model, the total lag stays around 4–6 seconds: the viewer already hears the next phrase while the screen still shows the previous one.

The first source of lag is window size. Until enough context is accumulated, streaming speech recognition cannot parse the phrase confidently: a short window produces errors at word boundaries, while a long one adds waiting.

The compromise is to show an interim version and replace it with the final one after the phrase closes, but then the replacement mechanism itself becomes a separate engineering task.

The second bottleneck is hypothesis instability. A displayed word is suddenly replaced by another, the line “jumps,” and text-audio synchronization visually breaks even when technically everything is fine.

Here a fixation policy helps: a point is defined after which the line is considered immutable, and only its tail is rewritten.

The third layer is infrastructure and audio quality. Processing queues, network delays, noise, echo, and overlapping voices reduce accuracy, while trying to wait for a cleaner fragment adds time.

A separate challenge is multiple speakers and translation: speaker turn labeling adds to the lag. Below is how we fit the latency budget into two seconds and preserve readability.

What changes after implementing real-time captions

The streaming recognizer outputs text as speech arrives, not after it ends. Caption latency relative to voice is reduced to 1–2 seconds versus 5–8 with batch processing — the viewer reads and hears the same thought simultaneously, rather than catching up with the picture.

Accuracy rests not on a universal model but on fine-tuning for a specific stream: company names, job titles, and industry terms go into the dictionary and stop being recognized “by ear.” We combine Whisper and open acoustic models into one loop where noise reduction runs before recognition — hall hum, ventilation, and lavalier mics no longer knock out entire phrases.

Speaker changes stop tearing the feed: voice separation adds a “Speaker 1 / Speaker 2” label, utterances are not glued into one, and the share of lost fragments in live dialogue drops from 10–15% to 2–3%.

Infrastructure load is relieved by queuing and horizontal scaling: peak hours are distributed across several recognition nodes, and latency does not grow as the number of streams increases.

Instead of manual retyping, the editorial team maintains the glossary and punctuation rules — its time goes into terminology, not verbatim typing.

Caption output methods: comparison of approaches

Caption latency is determined not by the recognition model but by where the text is computed and how it reaches the screen. Each route has its own balance: response speed, resilience to poor connectivity, and speech privacy requirements. Below are three caption output modes that are actually chosen between at the start of a project.

In practice, the choice comes down to two questions: whether audio can leave the perimeter and how many viewers are watching the broadcast simultaneously.

After that, modes are combined — some channels compute locally, some go to the server, and switching is done on the delivery side without rebuilding the player.

How we implement Live Captions: work stages

Before changing anything in the broadcast, we understand how it is structured: where the audio comes from and where fractions of a second are lost. Caption latency rarely originates in recognition itself — it is accumulated by buffers, transcoding, and network jitter. That is why each stage below ends with a measurement, not a report.

1. Broadcast stream audit. We capture real tracks, examine the spread of latency and loss, and behavior under channel overload and camera switching.

   We record where the audio is already ready for recognition and where segmentation based on an individual calculation is neededRecognition setup. We select a model for the language and acoustics, add a dictionary of terms and names, and enable punctuation and speaker labeling. Noise and echo are cleaned separately — they account for most errors in live broadcast.

2. Pipeline assembly and latency measurement. We connect audio capture, recognition, and caption output, and set a buffer that damps network jitter but does not accumulate latency. The goal is to fit within two seconds from word to caption on screen.

3. Test broadcast. We run the solution on a live broadcast: several viewers at once, different channels, pauses, and switches. We look at where gaps appear and adjust behavior during failures.

4. Fault tolerance. We check that when the stream is lost, captions do not disappear silently but switch to a clear state, and that after recovery the system catches up with the broadcast without manual intervention.

5. Solution acceptance. We compare metrics against the agreed ones, run failure scenarios, record settings, and hand over to operations: diagrams, dictionary update policy, and instructions for the on-call shift.

Documentation and policy package after implementation

Implementation is not closed at the moment captions first go on air, but when your team can bring up, check, and repair the loop without us.

That is why the package includes working documentation: with it, the on-call engineer finds the cause of desynchronization or accuracy drop in minutes instead of piecing the picture together from chat.

Each item is agreed upon at acceptance — there is a check for it, not just text. The package remains with you and is updated as the loop changes.

• Stream diagrams — the path from signal source to caption output, with a backup route and markings of points whose failure the viewer notices first.
• Operations documentation — component composition, operating parameters, and load limits within which latency stays within two seconds.
• Operating policy — the normal cycle: startup, switch to backup, planned shutdowns, maintenance windows, and the loop owner on the customer side.
• Access policies — roles and rights: who changes recognition settings, who edits dictionaries, who sees broadcast logs.
• Editorial instructions — how to add terms, names, and titles to dictionaries, how to mark disputed fragments, and where to report recurring errors.
• Failure procedure — a decision tree from “captions lag” to “stream is not recognized”: what to check first and when to switch to backup.
• Acceptance protocol — a checklist of checks with actual latency and accuracy measurements on your broadcasts, not on a test bench.

Case study: live captions with latency up to two seconds

Before the work

The broadcast ran without synchronous captions: the transcript was assembled manually after the performance and published as a separate text.

The first working version in live broadcast lagged behind speech by 8–12 seconds, and the share of correct words on terms and speaker names held around 80% — captions formally existed, but they could not be used.

What we rebuilt

The task was divided into three layers: audio stream ingestion with a controlled buffer, recognition with interim hypotheses, and dictionary-based post-processing. Base recognition was built on Whisper, and the stream was cut by pauses so that long phrases would not delay output.

The main change was not waiting for the end of a sentence: a hypothesis is output immediately and refined, and the last words are rewritten if the interpretation changes. That is why latency stopped depending on phrase length.

Before the broadcast, a dictionary of names, abbreviations, and proper names is loaded, and speakers with products no longer turn into a random set of words.

Acceptance metrics

Measurements were taken on a recording of a real broadcast:

• output latency: from 8–12 seconds to 1.4–1.9 seconds;
• share of correct words: 80% → 94% on general vocabulary, 97% after the terminology dictionary;
• missed phrases: zero over a multi-hour recording with speaker changes;
• stream interruption: captions recover on their own, the broadcast does not need to be restarted.

The implementation case showed that quality rests not on the model but on the buffer, dictionary, and rewrite logic. For the customer, we deploy this as a service with metric-based acceptance, not as a one-time setup.

How streaming speech recognition differs from batch transcription

Batch transcription works on accumulated recording: the engine sees the whole array and can rewrite what has already been recognized, relying on subsequent context. Streaming recognition receives audio in fragments and must output text before the end of the phrase is known.

The difference is not in the model but in the amount of context available at the moment of decision.

Transferring the batch approach directly to live broadcast does not work. Decoding runs in overlapping windows, each window is recomputed from scratch, and latency grows with window length — while accuracy depends precisely on long context.

Bidirectional models need a lookahead buffer, an artificial wait that accumulates on every fragment. On an hour-long recording this is unnoticeable; in live broadcast it decides everything.

The streaming scheme changes the computational load profile: instead of peaks on long windows, there is an even budget per fragment and interim hypotheses. The tail of a phrase is rewritten as data arrives, and a stabilizer keeps the text from twitching.

Models of the Whisper class are strong on large windows, so in a stream they are constrained and supplemented by a fast primary recognizer.

The boundary of applicability runs along the tolerable time until text appears. Archives, meeting minutes, long interviews — here batch transcription is more accurate and more efficient in load.

Live event, captions for the hard of hearing, operator prompts — only streaming mode: the decision is needed while the phrase is still sounding.

The choice of approach is dictated not by model size but by the allowable pause between an utterance and the word on screen, together with the computational load profile.

What to check before launching real-time captions

Before launching real-time captions, they check not the recognition quality on a single utterance but the behavior of the entire loop under load.

Below are the questions that most often come up at acceptance: noise in the hall, connection loss, transcript privacy, translation, and system behavior over multi-hour broadcasts.

How the system behaves when the stream breaks

Latency under two seconds is achieved through interim hypotheses: a draft line is visible immediately, and the final version arrives after the phrase closes.

If the channel drops, the client holds a local buffer and sends it after reconnection — the viewer sees a gap of a couple of lines, not the loss of a chunk of the broadcast.

Transcript privacy: where the texts live

Transcription can run on our loop, without sending data outside, or with masking of personal data before storage. We separately record whether the text is stored after the broadcast and after how long it is deleted — by default, recording is disabled.

Caption translation: where quality is lost

Names, terms, and product names break first, so an exception list is added to the glossary in advance. Translation adds its own rephrasing step to latency rather than doubling it — but only in a streaming scheme, not when translating finished paragraphs.

Long broadcasts: how to avoid degradation

Sessions are rotated, and context is limited to a window of the latest utterances — otherwise the system slows down toward the end of the stream. At acceptance, we run a broadcast several hours long and look at latency stability and the share of recognized words by the finale.

Let's discuss the task: what we clarify on the first call

So that the timeline estimate relies on facts rather than assumptions, we structure the first call as gathering inputs for your scenario.

In one conversation, we review the audio stream, infrastructure, and integration points, after which we return with an implementation plan and stage boundaries.

On the first meeting, we clarify:

• where we take audio from: broadcast line, web conference, studio console, or camera stream — the capture scheme and input losses depend on this
• speech composition: languages and dialects, industry jargon, names and terms that need to be added to dictionaries
• where we display captions: separate screen, overlay on the broadcast, personal account, or transfer to your system
• target latency and its priority relative to recognition accuracy specifically on your material
• how many simultaneous broadcasts and viewers the solution must handle during peak hours
• deployment constraints: own loop, private cloud, or external services, and who is responsible for access
• who on your side accepts the result and by what criteria considers the check successful

We test hypotheses with a pilot on one broadcast: it shows real latency, the share of correct recognitions, and system behavior under noise without slowing down other tasks.

Describe the task in two or three paragraphs — we will return with an estimate, implementation plan, and clear timelines.