STT from video: where accuracy and time are lost
STT from video on a clean studio recording with one speaker produces almost finished text.
As soon as it is a street interview, a call with echo, or a webinar with two presenters, recognition errors grow: names, abbreviations, numbers, and product names turn into a meaningless set, and video transcription stops saving time.
A separate issue is subtitle timecodes. The model may hear a word correctly but place an utterance boundary in the middle of a phrase, stretch a caption, or shift it relative to sound.
The viewer sees desync, and the editor manually fixes timing across the entire track — on hour-long material, that is hours of work not included in the plan.
The third source of failures is overlapping speech and speaker changes. Without diarization, utterances merge into one stream, and at phrase intersections the model attributes words to the wrong person. Correction turns into manual comparison with the recording.
Accuracy is raised not by replacing the model but by a processing chain: noise reduction and level alignment, speech activity labeling, diarization, a dictionary of terms and names for a specific project, and speech recognition confidence scoring.
Only low-confidence areas go to manual proofreading — fractions of a percent, not the entire text. We explain how this is addressed.
What automatic archive transcription provides
Speech in an archive is the densest carrier of meaning and the most expensive to process: an hour of recording requires several hours of manual operator work.
Automatic transcription removes this ratio: 5–10% of the source time goes to proofreading, and the rest goes to disputed fragments, noise, and voice overlaps, where the model itself marks low confidence.
Time to a finished transcript stops depending on archive volume: a recording several days long is processed overnight, and the result appears immediately with timecodes and speaker labeling.
Then speech search works over the text — the needed fragment is found by phrase, not by the author's memory, and opens in the recording at the exact second.
A separate class of failures that goes away with manual labeling: broken subtitles, lost segments at junctions, different names for the same speaker. Video subtitles are assembled from the same transcript and exported to SRT — timecodes are tied to the track, not set by eye.
Archive transcription becomes not a one-off project but a service: new recordings are added to a shared index, old ones do not require repeated manual labeling, and quality is confirmed by spot-listening to control fragments.
Video processing modes: batch and streaming
The choice of scheme is determined not by archive size but by whether the result is needed at the moment of speech.
Batch video processing takes the file as a whole: recognition runs across the entire video, the model sees context before and after the phrase, so accuracy on noisy tracks, when the speaker changes, and when languages switch is higher. The price for this is delay: the transcript appears after the file is uploaded, not live.
Streaming recognition cuts audio into short windows and outputs text with a delay of fractions of a second — this is how subtitles for webinars and call analysis work.
The limitation is the opposite: the model has no access to further context, it restores names, terms, and slips worse, and on an unstable connection, windows are lost.
We put both modes on one core (models of the Whisper family and their analogues), so switching does not require a second integration and retraining on your data.
| Scheme | When it fits | Limitations |
|---|---|---|
| Batch | Meeting recordings, interviews, courses, video archive | No real-time text |
| Streaming | Live subtitles, call analysis | Lower accuracy on terminology, sensitive to connection |
| Hybrid | Webinars and calls recorded for later | Both pipelines and a reprocessing queue are needed |
The hybrid scheme covers both scenarios: the stream provides draft subtitles live, and after recording ends, the same file goes through a batch run, and the draft is replaced with an accurate transcript.
We deploy the local pipeline where the recording must not leave your infrastructure: the same set of models runs on your infrastructure, and only processing metrics go outside.
Stages of speech recognition implementation
We structure the work so that quality is measurable, not "by ear". First we fix a baseline on real recordings, then configure processing and check the result on the same fragments — and only after that do we hand over the support procedure.
- Recording audit: we analyze the collection — sources, channels, container formats, share of noisy and multichannel files, languages and accents, track lengths. The output is a data map and an understanding of where preprocessing is needed.
- Control sample annotation: we manually transcribe several dozen fragments, including overlapping speech and poor recordings. This is the baseline word error rate and at the same time the acceptance set.
- Voice diarization and timing: we separate utterances by speaker, set timecodes, process overlaps and short insertions. The reviewer gets not continuous text but a conversation with attribution.
- Dictionary and normalization: we add names, abbreviations, product names, and domain terms, and bring numbers, dates, and punctuation to a unified form — this step removes most errors in business vocabulary.
- Output assembly: subtitles with timecodes, speaker-attributed text, a keyword search index, and links to moments in the recording.
- Acceptance on the control sample: we repeat the measurement on the same set, compare with the baseline, analyze remaining discrepancies, and decide what to improve with the dictionary and what with the processing scheme.
- Support procedure: we describe how to add terms, update the dictionary when new topics appear, and restart archive processing as volumes grow.
This sequence addresses the main risk of speech recognition implementation — when the system works on a demo recording and falls apart on the production archive.
Recognition configurations, dictionaries, and procedures
After speech recognition implementation, the client's team is left not with a closed service but with a set of artifacts: with them, the system's behavior can be changed independently, without waiting for us.
Everything is stored in the client's repository and versioned together with the code — changes go through normal review, not approval by correspondence.
- Pipeline configuration — declarative files with parameters for splitting the stream into channels, cutting into fragments, speech detection thresholds, and pause handling rules. Editing one value changes the behavior of the entire pipeline without rebuilding.
- Term dictionary — a domain glossary: product names, job titles, abbreviations, place names. It is connected to recognition as a hint and is updated by the client with a regular text file.
- Access policies — differentiation by recordings, roles, and retention periods: who exports the full transcript, who sees only subtitles, what is deleted automatically.
- Subtitle templates — rules for line breaks, display duration, margins, and export profiles for different platforms.
- Quality procedure — spot checks: the share of transcripts going to manual review, reference recordings, and the word error rate threshold below which output is not accepted.
- Model update procedure — how to compare a new model with the working version on the reference set, when an update is acceptable, and how to roll back if quality drops.
- Operating instructions — where to view processing logs, what typical failures mean, who receives notifications, and how to restart processing for a specific recording.
We walk through each artifact with the client's team during handover: we show what it affects and assign an owner.
Case study: archive of interviews and lectures with subtitles
The client came with an archive of interviews and lectures — more than 300 hours of recording. Some files were from a lavalier mic in a noisy room, others from two microphones with overlapping speech.
Transcription was manual: an editor spent about six hours per hour of recording, subtitles for lectures came out weeks after publication, and half the archive remained without text and could not be found by search.
How we built the pipeline
First we ran the audio through normalization and pause-based segmentation, then speech recognition on the Whisper model with diarization: utterances are split by speaker, and timecodes stay within a fraction of a second.
A separate step is the glossary: names, abbreviations, and industry terms that the model hears incorrectly by default. After recognition comes post-processing — punctuation, utterance segmentation, export to SRT and VTT with manual editing in a single window.
Metrics before and after
Quality was measured on a manually verified control sample of 20 files — by word error rate and timecode accuracy.
On clean recordings, the error rate dropped from 27% to 6%; on noisy ones, from 40% to 14%; the share of segments the editor edits manually became 12% instead of all. A 40-hour archive was processed overnight: an hour of recording takes a few minutes.
Subtitles for lectures are now published together with the video, and the archive text became searchable by phrases within utterances.
How transcription differs from subtitles and a protocol
Transcription, subtitles, and a text protocol are three different products assembled from the same speech recognition. Transcription preserves everything said: a continuous stream of utterances with time marks, speaker labels, slips, and repetitions.
Subtitles are segmented text tied to frames and trimmed by line length so the viewer can keep up. A protocol is a semantic summary without timings: decisions, assignments, agreements.
The divergence begins after the acoustic model. Recognition outputs a stream of words with time marks and confidence scores — this is the shared input for all three variants.
Then different post-processors work: for transcription, punctuation, normalization of numbers and dates, and voice separation matter; for subtitles, splitting into short utterances, semantic line breaks, and control of display time in frame.
The subtitle format sets hard limits: how many characters are allowed per line, how many lines per frame, and what the minimum and maximum display duration is.
Hence the standard SRT and WebVTT for players and publishing platforms, as well as tables with word-level annotation for editing. In multilingual videos, translation synchronization with the source timing is added — otherwise utterances start to drift relative to speech.
A text protocol is assembled differently: repetitions and slips are removed, terms are brought to a single dictionary, utterances are labeled by participant, and key points are placed in a separate block.
This is where the domain glossary and proofreading are connected — without them, a meeting transcript remains raw text that cannot be used to make a decision.
The choice of output format is determined by the consumer, not the technology. An editor needs subtitle files and word-level marks, a support service needs a search index over the transcript, and a manager needs a protocol with assignments.
One recognition run serves as the basis for all variants: then post-processing and export rules are configured, so the set of formats expands without re-transcribing the recording.
Technical concerns about recognition quality
Concerns about recognition come down to three things: accuracy on real material, the fate of source recordings, and the amount of manual refinement after transcription.
None of these is addressed by the promise that "the model will understand everything" — each needs a measurable criterion fixed before start.
Therefore, before work we measure quality on a control sample from your recordings: we calculate word error rate, voice labeling quality, and losses from noise, accent, and overlapping speech. These same figures become the acceptance bar for the result.
How accurate will recognition be on our material?
Accuracy is determined not by the engine but by acoustics and terminology. We classify errors on the control sample by category and show where the cause is noise and where it is specific words.
After track cleanup, voice separation setup, and dictionary configuration, the error rate drops predictably, and this level is fixed as the acceptance criterion.
Where do recordings go and who has access to them?
Audio and transcripts are processed in an isolated environment — yours or our dedicated one — without export to external services. Access is role-based, source files are deleted according to an agreed procedure, and actions are logged.
For sensitive materials, we deploy processing inside your infrastructure, and data does not leave the perimeter.
How many edits will remain for the editor?
The amount of refinement is determined by the industry-specific dictionary and role labeling. We collect terms, abbreviations, product names, and names, and add context-aware replacements so abbreviations do not break neighboring words.
Subtitles are delivered with timecodes and text in the required formats — editing comes down to proofreading disputed fragments, not rewriting.
Let's discuss the task of recognizing your recordings
So that the project estimate is based on your data rather than average figures from general reports, send a task description and several sample recordings.
They are used to calculate the labor intensity of audio cleanup, model tuning to your vocabulary, and pipeline assembly — and we return with an estimate and work plan.
- Pilot sample: 3–5 recordings of 5–15 minutes — a quiet room and a noisy room, a microphone and a phone line, one speaker and a conversation of several people.
- Language