Automatic video subtitle generation based on Whisper large-v3

Why automatic subtitle generation breaks on real videos

Automatic subtitle generation works confidently on a demo recording in a quiet room and falls apart on the first real material. In the track, there is music, ventilation noise, conference room echo, and microphones with different loudness levels for participants.

The speech recognition model, including the current Whisper branch, outputs text where a person hears one word, and produces recognition errors where the recording is objectively clean: two voices overlap mid-word, and the output is a merged line attributed to the wrong speaker.

The second class of failures is vocabulary. Industry jargon, names of internal systems, abbreviations, and Latin characters inside a Russian phrase are recognized "by sound" and turn into a meaningless set of letters.

A separate story is language switching within a recording: a participant switches to English for two sentences, and then video-to-text transcription proceeds with the wrong language model until the file is manually segmented.

The third class is volume. Subtitles for a video archive are needed not for one clip: it is dozens of hours of lectures, webinars, and meeting recordings.

Manual transcription of recordings by an editor gives acceptable quality, but runs into person-hours — an hour of recording takes several hours of work, and the archive grows faster than it can be processed. When the queue is measured in months, the task quietly stops moving.

We break down how this is solved: where to fix the input audio, where to connect a dictionary and voice separation, and where to build a batch archive processing pipeline.

Effect of implementation: speed, single standard, coverage

Transcription with Whisper is faster than real time: a one-hour episode becomes a finished track in minutes, not in an editor's shift.

Speech recognition accuracy on studio sound is sufficient for publication without edits, while on interviews with noise and accents, proofreading remains — but targeted, by highlighted fragments, not across the entire text.

On the pilot catalog, the gap by stages looks like this:

A single track format removes a whole class of manual input errors: discrepancies in timecodes, lost lines, drifting punctuation.

When markup is aligned across the entire catalog, subtitle search, auto-translation, and vertical version cutting begin to work across the whole archive, not only on the latest publications.

This is exactly what the value of automatic subtitle generation rests on: transcription stops being the release bottleneck.

Ways to get subtitles: batch, streaming, hybrid

The operating mode is chosen not by preference, but by where the video comes from and what is needed at the output.

Archives, live broadcasts, and individual fragments require different queue organization: in some cases throughput matters, in others latency, and in others it is enough to transcribe ten minutes from a three-hour recording.

Batch processing queues files and returns finished tracks in a batch, streaming transcription returns text as speech proceeds, and between them lie two compromise scenarios. Below is how they differ in practice.

Choosing a mode is always a trade-off between latency and accuracy: a streaming pipeline makes more mistakes on names and terms than a batch one, but returns text immediately.

In practice, pipelines are combined: the live broadcast runs in streaming, and the recording after the broadcast is run in batch on the speech recognition model and replaces the draft track. Let's break down how this is assembled into one process.

How implementation is built: from archive audit to handover

Transcription implementation begins not with launching a model, but with an inventory of the archive: total duration of the collection, languages and accents, share of overlapping speech and music, quality of audio tracks, availability of ready texts and edit lists.

These inputs determine both the processing profile and the volume of manual proofreading — therefore a video archive audit comes first, rather than being discovered on the first batch.

1. Audit of the video archive and requirements. We categorize the collection by category and sound quality, and mark where manual proofreading is needed. Here we also agree on formats: embedded subtitles, separate files, speaker markup.
2. Processing scheme. We determine silence-based segmentation, channel order for multitrack recordings, and system behavior on music and unrecognized fragments.
3. Setup for recording types. For interviews, webinars, lectures, and outdoor shoots, we set our own segmentation profiles: uniform settings produce different quality across genres.
4. Project terminology dictionary. Names, abbreviations, product names, professional vocabulary. This is the fastest way to raise accuracy without retraining the model.
5. Control sample of recordings. We measure the share of correct words, correctness of time labels, and line length. Disputed cases are analyzed before setup, not after delivery.
6. Track acceptance procedure. We fix measurable criteria: permissible error share by category, line length, rate of change, pauses. By this, you accept the work component by component.
7. Handover. We provide configurations, dictionaries, and instructions, and show how to run processing of new recordings on your own.

After handover, a working pipeline and a clear acceptance procedure remain — new materials go through automatic subtitle generation the same way, without our involvement.

Dictionaries, configurations, and output markup rules

Automatic subtitle generation based on Whisper from the current branch ends not only with finished files, but also with a set of artifacts your team can use to run new recordings itself.

Configurations, dictionaries, and markup rules live in the project repository, so the next material is transcribed with the same settings — without us.

• Configuration profiles for typical scenarios. Separate parameter sets for interviews, lectures with presentations, and recordings with multiple speakers: line length, pause before a new track, behavior when speech overlaps.
• Project terminology dictionary. Product names, abbreviations, speaker names, and spelling variants. It is connected before generation, so the error share on narrow vocabulary drops without manual proofreading.
• Rules for track markup and time segmentation. Line boundaries are tied to semantic groups, not to character count: minimum caption duration, line wrapping, pause splitting.
• Transcription documentation. Export formats — SRT and VTT, encoding, track numbering, matching voices to speakers, source storage.
• Editor instructions. What must be corrected, which discrepancies are acceptable, how to mark disputed places and update the dictionary so that edits work on subsequent runs.
• Result acceptance procedure. Control sample, norm for the share of words with discrepancies, acceptance stop points until issues are resolved.
• Re-run procedure. How to update a profile or dictionary and regenerate subtitles without reassembling the video.

Case: from manual proofreading to streaming archive transcription

The media library came with an archive of approximately 900 hours: interviews, lectures, conference recordings. Transcription was done manually — one editor listened and typed the text, another proofread terms and name spelling.

One track took six to eight hours, so subtitles were released selectively: for approximately every third recording.

How processing was organized

Files enter a queue, Whisper in a large model configuration returns text with time labels, and a normalization layer brings numbers, abbreviations, and name spelling in line with the project dictionary. Long segments are cut by pauses so a line does not occupy half the screen.

Then automatic checking compares text length with sound, looks for empty intervals and disputed vocabulary — the editor opens only marked places, not the entire track.

What the result was

The share of tracks published without edits grew from 31% to 86%. The release time for one track decreased from six hours to 35 minutes: subtitles come out together with video publication, not a week later.

The archive was processed in three weeks, and then the flow continues continuously — new recordings receive transcription on the day of upload.

Transcription sits next to the video as a text layer, so search across the media library works by lines, not only by file names.

How does streaming transcription differ from batch transcription?

Streaming transcription cuts audio into short windows and returns text as it arrives: transcription latency is measured in seconds, but the model sees only limited context.

At segment boundaries, punctuation breaks, names and terms are lost, and overlapping speech from two speakers collapses into one line. Speed here is paid for with quality: the text is suitable for reading, but not for publication without editing.

Batch processing receives the file as a whole: before recognition, the current-branch Whisper model works with the full recording context and timecodes.

On complex materials — noise, accent, language switching within a line, several voices in one channel — a usable result is delivered specifically by a batch run. The price for this is waiting: subtitles are ready not at the moment of speech, but after processing finishes.

Each scheme breaks in its own way: streaming cannot handle interviews with interruptions and captions over speech, while batch is useless for live broadcast where subtitles are needed at the moment of conversation. Therefore the mode is chosen not by file length, but by the requirement for when the text appears.

A hybrid processing mode removes this contradiction: a draft track runs in streaming with a delay of seconds, and after publication the recording is run in batch — with hot words and a terminology dictionary — and the final text replaces the draft by timecodes, without changing the link or player settings.

We select the mode by content type: live broadcast — streaming, course archive or interview — batch, event with recording — hybrid.

Technical concerns: quality, languages, access

Most questions about automatic subtitle generation are not about the speech recognition model Whisper itself, but about how it behaves on a real archive.

Below are the concerns that usually come up before launch: speech recognition quality, language support, and access to video recordings.

What should be done with mixed languages in one recording?

Language is determined by fragment, not by the entire file, so short insertions in another language are recognized in their own language branch.

Names, abbreviations, and industry terms go into a replacement dictionary, which we assemble from your glossary — they account for the main share of edits.

Do you need access to the source video recordings?

We process a copy of the archive on our pipeline or deploy everything inside your perimeter — then the sources do not leave the infrastructure. Read access to video and write access for subtitles are needed; we issue them to a separate service account with a limited term.

We have a partially annotated archive — will this help or hinder?

It will help. If some recordings already have subtitles, we use them as a reference: compare with automatic markup, calculate the match share, and configure formatting rules. A tenth of an annotated archive is usually enough to improve quality on the remaining recordings.

What happens after editor edits?

Edits are stored as a separate layer over the machine output. After changing the glossary or settings, we regenerate only the affected sections, not the entire archive; version history and timings are preserved, and rollback is one operation.

Let's discuss the task: we will return with an estimate and plan

To make the transcription project estimate accurate rather than "plus or minus an order of magnitude," we need a minimum of inputs. Below is what to send to start the discussion: this is enough to calculate resources and propose an implementation plan.

• Volume and types of recordings: how many hours per month, container formats, how many tracks in each recording.
• Examples of complex fragments: 3-5 pieces where speech overlaps, with many terms, names, or background noise.
• Requirements for tracks: which track is primary, how to separate voice from music, whether a mix and dubbing are needed.
• Languages: primary language, insertions in other languages, whether on-the-fly language markup is needed.
• Markup requirements: subtitle file format, line length, characters per second, styles, whether speaker labels are needed.
• Access restrictions: where recordings are stored, whether they can be exported outside, who views the result and how.
• Acceptance criteria: how quality is measured (error share) and who performs proofreading.

In response, we do not return with general words: a transcription project estimate by volume, a draft implementation plan, and track requirements for your material.

The current-branch Whisper family model covers the main part of the work, but accuracy rests on audio preparation — therefore examples of complex fragments matter most.

Describe the task — we will return with an estimate and plan.