Implementing Speech Endpointing for Voice Bots

We've faced this situation: a voice bot cut off the customer mid-phrase because the silence threshold was too strict. Or conversely—it hung for 3 seconds, creating awkwardness. Both cases result from poor implementation of speech endpointing (end-of-speech detection) and suboptimal VAD. In this article, we'll walk through how to configure VAD, pick thresholds, and build adaptive endpointing that works for different scenarios.

Problems We Solve

False triggers occur due to too short a silence threshold (<500 ms) or low VAD sensitivity. The user pauses, but the system already sends a request. This is especially critical in contact centers: the bot interrupts, the agent gets annoyed. The cost of such an error is a lost customer.

Missed end-of-utterance—the opposite situation: a high threshold (>1500 ms) or VAD "doesn't hear" the end of speech against background noise. The dialogue drags on, and the user loses patience. Our experience shows that 80% of issues are solved by choosing the right VAD and adapting thresholds to the scenario. Savings on re-engineering—up to 40% of budget.

Processing latency: VAD must work in real time, with p99 latency <100 ms. We use Silero VAD [Silero VAD paper] in ONNX Runtime, or WebRTC VAD (lightweight but worse in noise). For high-load systems—batching on GPU.

How to Choose the Silence Threshold for Different Scenarios

For a phone voice bot, optimal parameters: silence 600–800 ms, minimum speech 200 ms. For dictation: silence 1500–2000 ms. For smart home (quiet background): 500–600 ms. We always test on real recordings with noise. An adaptive approach delivers UX gains: on open questions, the threshold increases; on commands, it decreases.

How We Do It: Stack and Implementation

We use Python 3.11, PyTorch 2.2, ONNX Runtime 1.17, Silero VAD v4.0. For asynchronous processing—asyncio. Here is a basic detector implementation (used in production):

import collections import time from enum import Enum class SpeechState(Enum): SILENCE = 0 SPEECH = 1 class EndpointDetector: def __init__( self, vad, sample_rate: int = 16000, frame_ms: int = 30, silence_threshold_ms: int = 700, # pause for termination min_speech_ms: int = 300, # minimum utterance length ): self.vad = vad self.sample_rate = sample_rate self.frame_bytes = int(sample_rate * frame_ms / 1000) * 2 self.silence_frames_needed = silence_threshold_ms // frame_ms self.min_speech_frames = min_speech_ms // frame_ms self.state = SpeechState.SILENCE self.silence_counter = 0 self.speech_buffer = bytearray() self.speech_frame_count = 0 def process_frame(self, frame: bytes) -> tuple[bool, bytes | None]: """ Returns: (endpoint_detected, speech_audio_or_none) """ is_speech = self.vad.is_speech(frame, self.sample_rate) if is_speech: self.state = SpeechState.SPEECH self.silence_counter = 0 self.speech_buffer.extend(frame) self.speech_frame_count += 1 else: if self.state == SpeechState.SPEECH: self.silence_counter += 1 self.speech_buffer.extend(frame) # include trailing silence if self.silence_counter >= self.silence_frames_needed: if self.speech_frame_count >= self.min_speech_frames: audio = bytes(self.speech_buffer) self._reset() return True, audio else: self._reset() return False, None def _reset(self): self.state = SpeechState.SILENCE self.silence_counter = 0 self.speech_buffer = bytearray() self.speech_frame_count = 0 

In real dialogues, adaptive endpointing is needed. We use a classifier based on Intent Detection (e.g., via a small model like DistilBERT) that determines the request type and dynamically changes the threshold. Adaptive endpointing handles open questions 2x faster than a fixed 700 ms threshold.

# Different thresholds for different request types THRESHOLDS = { "open_question": 1200, # ms silence "yes_no": 500, "command": 600, "default": 700, } 

VAD Solutions Comparison

Choosing a VAD is a trade-off between accuracy and resources. For contact centers we recommend Silero, for IoT—WebRTC. Latency p99 is critical for voice bots: if it exceeds 100 ms, the dialogue becomes unnatural.

Work Process for Endpointing

1. Analysis—collect dialogue recordings, measure current metrics (latency, errors).
2. Design—select VAD (usually Silero), set threshold configuration, decide on adaptive classifier.
3. Implementation—integrate detector into voice stream (WebRTC or custom). Add monitoring via MLflow.
4. Testing—A/B test on 10% of traffic, compare with current solution.
5. Deployment—containerization, run on CPU nodes (Triton Inference Server). Team training.

What's Included in Turnkey Work

• Documentation—architecture description, parameters, monitoring instructions.
• Code—Python module with VAD, adaptive threshold, error handling.
• Test bench—simulator with real recordings.
• Training—call with team, Q&A.
• Support—2 weeks after deployment (bug fixes, load tuning).

Timeline: basic implementation—2-3 days, adaptive with ML—1 week. Cost is calculated individually, but such an upgrade pays off in 2-3 months by reducing pauses and increasing conversion. Proper endpointing configuration can cut operational costs by 20-30%.

Our experience: over 5 years working with voice assistants, 30+ successful projects. We guarantee stable endpointing operation on noisy lines. To evaluate your project, contact us—we will analyze your recordings and offer the optimal solution.

How to Avoid Mistakes When Implementing?

• Don't copy thresholds from one scenario to another: testbed must include your real audio (with noise, varying loudness).
• Document metrics: latency p99, false positive rate, false negative rate. Without them you won't know if it improved.
• Use an adaptive approach: even a simple threshold change by request type improves UX by 30%.

Get a consultation: contact us—we will evaluate your project and propose a solution.