Real-time Streaming Speech Recognition (STT)

We have faced the challenge: a call center with 50 operators needing real-time subtitles for supervisors. Batch STT had a 5-second delay, missing important dialogue moments. For live conference subtitles, a 2-second delay is already unacceptable, and in voice assistant STT, every extra millisecond degrades UX. The solution is real-time streaming speech recognition with partial results over WebSocket. Over 5 years, we built an architecture that maintains 100–500 ms latency under any load.

What Problems Does Streaming Speech Recognition Solve?

• Latency: Without partial results, the user waits for the end of the phrase. Streaming outputs preliminary transcription every 200–400 ms. For a call center, this means instant reaction – the supervisor sees the text 200 ms after it is spoken.
• Pauses and overlaps: VAD endpointing correctly handles silence and overlapping speech. Setting aggressiveness=2 cuts 90% of pauses without loss of meaning.
• Real-time accuracy: Low-latency models (Deepgram Nova-2) achieve WER <5% even at 200 ms. The cost of Deepgram Nova-2 is $0.0043/min, 40% cheaper than Google STT. For 1000 hours per month, self-hosted faster-whisper on GPU costs about $60 (GPU rental), saving $2,520 per month compared to Deepgram.

Streaming Speech Recognition Implementation

A typical production architecture we have deployed:

Microphone → WebSocket (WSS) → FastAPI → STT Engine → NLP → Response 

Key components are implemented in Python with async sockets, utilizing buffering optimization and endpointing algorithms.

WebSocket server on FastAPI

from fastapi import FastAPI, WebSocket from faster_whisper import WhisperModel import numpy as np import asyncio app = FastAPI() model = WhisperModel("medium", device="cuda", compute_type="float16") @app.websocket("/stream") async def stream_stt(websocket: WebSocket): await websocket.accept() audio_buffer = bytearray() try: while True: chunk = await websocket.receive_bytes() audio_buffer.extend(chunk) if len(audio_buffer) >= 32000 * 2: # 2 sec @ 16kHz 16-bit audio_array = np.frombuffer(audio_buffer, dtype=np.int16).astype(np.float32) / 32768.0 segments, _ = model.transcribe(audio_array, language="ru") partial_text = " ".join([s.text for s in segments]) await websocket.send_json({"type": "partial", "text": partial_text}) audio_buffer = bytearray() except Exception: await websocket.close() 

VAD (Voice Activity Detection)

We attach VAD before buffer accumulation: it cuts out silence, reducing the number of transcriptions.

import webrtcvad vad = webrtcvad.Vad(2) def is_speech(audio_chunk: bytes, sample_rate: int = 16000) -> bool: return vad.is_speech(audio_chunk, sample_rate) 

For endpointing, we maintain a sliding silence window of 500–800 ms.

Client side

const socket = new WebSocket('wss://api.example.com/stream'); const mediaStream = await navigator.mediaDevices.getUserMedia({ audio: true }); const recorder = new MediaRecorder(mediaStream, { mimeType: 'audio/webm;codecs=opus' }); recorder.ondataavailable = (event) => { if (socket.readyState === WebSocket.OPEN) socket.send(event.data); }; recorder.start(250); // 250ms chunks 

How VAD Improves Streaming STT

Without VAD, the engine processes the entire audio stream, including silence. This increases token cost and latency. In practice, we saw p99 latency increase by 30% without VAD preprocessing. Additionally, proper VAD endpointing reduces computational overhead by 40%.

How to Select the Right STT Engine?

The choice between cloud and self-hosted depends on load, confidentiality requirements, and budget. According to official Deepgram documentation, Nova-2 has 180 ms latency at p95. To illustrate cost savings: for 1000 hours of audio per month, self-hosted faster-whisper on GPU costs approximately $60/month (GPU rental), while Deepgram would cost $2,580/month, saving $2,520/month. These cost figures demonstrate that streaming STT can be cost-effective, especially with self-hosted solutions.

Self-hosted solutions save up to 80% at volumes >1000 hours per month. For multilingual projects, Google and Azure are preferable due to broader coverage.

Ensuring STT Latency Under 400ms

Key factors: choosing a low-latency engine, optimizing VAD endpointing, and configuring buffering. For self-hosted, we use faster-whisper with CUDA and INT8 quantization – this reduces latency by 30% without accuracy loss. Plus pre-segmentation of audio via VAD to avoid transcribing silence.

Key Metrics to Monitor

• STT latency p99 – no more than 400 ms for self-hosted, 300 ms for cloud solutions.
• CPU/GPU utilization – to avoid overload under peak load.
• WER (Word Error Rate) – tracked on a sample set.
• Number of active sessions – important for auto-scaling.

Turnkey implementation process

1. Analysis: Define language, number of speakers, expected RPS, endpointing requirements.
2. Design: Build flow diagram, select engine, VAD, and dispatching method.
3. Development: Code WebSocket server, integrate STT, configure auto-scaling.
4. Testing: Generate synthetic RTP streams, measure p99 latency, memory leaks.
5. Deployment: Deploy in k8s with Helm, connect monitoring (Prometheus + Grafana).
6. Handover: Documentation, team training, codebase with comments.

Deliverables

• Architectural diagram and reasoning for choice
• Repository with Docker containers and Helm chart
• API documentation (OpenAPI)
• Integration with client SDKs (Web, iOS, Android – optional)
• Load test plan
• 1 month support

Timeline and cost

Cost is calculated individually per task. Get a project estimate – contact us.

Our experience

We have implemented streaming STT for 10+ projects, from call centers to live subtitles. Our experience includes integration with deep dialogue frameworks and configuration for high load (up to 1000 concurrent sessions). We guarantee p99 latency < 400 ms for self-hosted solutions on NVIDIA A10G. Certificated in CUDA (NVIDIA).

We are ready to implement streaming STT turnkey. Get in touch for a consultation – we will discuss your task and choose the optimal architecture.