Fine-tuning Whisper for Domain-Specific Speech Recognition

Fine-tuning Whisper for Domain-Specific Speech Recognition

Your baseline Whisper large-v3 achieves 8–15% WER on general speech. But on specialized vocabulary in medicine, law, finance, or technical jargon, that number jumps to 25–40%. Fine-tuning on a domain dataset drops it to 3–8%. We fine-tune Whisper for your specific needs—from medical dictations to call center calls. With over 20 completed projects and 5+ years in speech recognition, we guarantee at least a 2× reduction in WER. Let's evaluate your project—contact us for a free initial assessment.

Why Fine-tune Whisper?

Cloud services like Google Speech-to-Text and Azure Speech offer generic models that don't adapt to narrow terminology. Fine-tuning on your own data delivers up to a 5× WER reduction compared to these APIs. For example, on medical dictations, fine-tuned Whisper achieves 6.2% WER versus 28% for Google Speech-to-Text—a 4.5× improvement. Plus, your audio stays on your own infrastructure—no risk of confidential data leaks.

Baseline Whisper struggles with rare terminology ("hepatosplenomegaly", "force majeure"), regional accents, noisy environments, and code-switching. Fine-tuning addresses all these.

Technical Implementation

Preparing a Domain Dataset

Minimum useful size is 10–20 hours; optimal is 50–100 hours. Use augmentation: street noise, reverberation, pitch change. Example preparation in Python:

from datasets import Dataset, Audio import pandas as pd def prepare_whisper_dataset( audio_dir: str, transcripts_csv: str, target_language: str = "russian" ) -> Dataset: """ transcripts_csv: columns = [audio_file, transcription] """ df = pd.read_csv(transcripts_csv) dataset = Dataset.from_dict({ "audio": [f"{audio_dir}/{f}" for f in df["audio_file"]], "sentence": df["transcription"].tolist() }) dataset = dataset.cast_column("audio", Audio(sampling_rate=16000)) return dataset 

Preprocessing with Whisper's feature extractor:

from transformers import WhisperFeatureExtractor, WhisperTokenizer feature_extractor = WhisperFeatureExtractor.from_pretrained("openai/whisper-large-v3") tokenizer = WhisperTokenizer.from_pretrained( "openai/whisper-large-v3", language="Russian", task="transcribe" ) def prepare_dataset(batch): audio = batch["audio"] batch["input_features"] = feature_extractor( audio["array"], sampling_rate=audio["sampling_rate"] ).input_features[0] batch["labels"] = tokenizer(batch["sentence"]).input_ids return batch dataset = dataset.map(prepare_dataset, remove_columns=["audio", "sentence"]) 

Fine-tuning with Seq2SeqTrainer (including LoRA)

[Full training code shown below. For GPU memory below 24 GB, we use PEFT/LoRA to train only 1% of parameters.]

from transformers import ( WhisperForConditionalGeneration, Seq2SeqTrainingArguments, Seq2SeqTrainer ) from peft import get_peft_model, LoraConfig, TaskType import evaluate import torch model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v3") model.generation_config.language = "russian" model.generation_config.task = "transcribe" model.generation_config.forced_decoder_ids = None # Apply LoRA if needed lora_config = LoraConfig( r=32, lora_alpha=64, target_modules=["q_proj", "v_proj"], lora_dropout=0.05, bias="none", task_type=TaskType.SEQ_2_SEQ_LM ) model = get_peft_model(model, lora_config) model.print_trainable_parameters() # trainable params: 15,728,640 || all params: 1,557,741,568 (1.01%) training_args = Seq2SeqTrainingArguments( output_dir="./whisper-finetuned-ru-medical", per_device_train_batch_size=16, gradient_accumulation_steps=1, learning_rate=1e-5, warmup_steps=500, max_steps=4000, gradient_checkpointing=True, fp16=True, evaluation_strategy="steps", per_device_eval_batch_size=8, predict_with_generate=True, generation_max_length=225, save_steps=500, eval_steps=500, logging_steps=25, report_to=["tensorboard"], load_best_model_at_end=True, metric_for_best_model="wer", greater_is_better=False, push_to_hub=False ) wer_metric = evaluate.load("wer") def compute_metrics(pred): pred_ids = pred.predictions label_ids = pred.label_ids label_ids[label_ids == -100] = tokenizer.pad_token_id pred_str = tokenizer.batch_decode(pred_ids, skip_special_tokens=True) label_str = tokenizer.batch_decode(label_ids, skip_special_tokens=True) wer = 100 * wer_metric.compute(predictions=pred_str, references=label_str) return {"wer": wer} trainer = Seq2SeqTrainer( args=training_args, model=model, train_dataset=dataset["train"], eval_dataset=dataset["test"], data_collator=data_collator, compute_metrics=compute_metrics, tokenizer=feature_extractor ) trainer.train() 

Results and Usage

Domain Results

Fine-tuning Whisper delivers a 3–6× improvement over the base model, outperforming cloud APIs that don't adapt to your domain vocabulary.

Inference with Fine-tuned Model

from transformers import pipeline asr = pipeline( "automatic-speech-recognition", model="./whisper-finetuned-ru-medical", device=0, torch_dtype=torch.float16, generate_kwargs={ "language": "russian", "task": "transcribe", "num_beams": 5 } ) result = asr("patient_recording.wav", chunk_length_s=30, stride_length_s=5) print(result["text"]) 

Our Service

We provide the full cycle:

• Audit of your audio data and target WER estimation
• Corpus transcription and augmentation (noise, reverberation, speed perturbation)
• Fine-tuning of the base model (Whisper large-v3, openai/whisper-medium, or distill-whisper)
• Experiments with LoRA, quantization, encoder architecture
• Testing on a held-out set and A/B comparison
• Model export to ONNX or TensorRT for inference
• Integration into your pipeline (REST API, Docker, batch processing)
• Documentation and training for your team

Process: Analysis → Design → Transcription → Training → Testing → Deployment

Timeline and Cost: Estimated timelines: from 2 weeks (for a 10-hour corpus) to 6 weeks (for 100+ hours with complex transcription). Typical budget starts at $5,000, with exact cost depending on data volume, target WER, and latency requirements. Get a consultation on Whisper fine-tuning for your task—we'll analyze the problem and find the best solution.

How to Get a Consultation?

Write to us via email or messengers—we'll analyze your task and select the most effective approach. We guarantee confidentiality and will sign an NDA.

Radford et al., "Robust Speech Recognition via Large-Scale Weak Supervision", 2022