Speech-to-Speech Voice AI Assistant: Development & Deployment

Problem: Dialogue latency kills UX

We've seen projects where the voice assistant took 3–4 seconds to respond — users simply abandoned the conversation. End-to-end latency is the key metric. Our experience shows: for a natural dialogue, you need to stay within 1.5 seconds from the end of user speech to the start of the response. We solve this with a Speech-to-Speech (S2S) architecture without text breaks. This architecture is critical for call centers, retail voice assistants, and medical systems — every second of downtime reduces conversion or even endangers patient health. Additionally, we implement latency monitoring by percentiles p50, p95, and p99 to guarantee stability even under load.

What problems does a Speech-to-Speech voice AI assistant solve?

The main technical challenges in the S2S pipeline:

• VAD + endpointing — detecting the end of a phrase with minimal delay (600–800 ms). Incorrect threshold leads to speech clipping or missing silence.
• STT latency — Whisper API gives 300–600 ms but adds network delay. Optimize via streaming mode and buffering.
• TTS streaming — synthesizing the first chunk in 200–400 ms, but the client must play on the fly. We use PCM stream with preloading.
• LLM reasoning — GPT-4o-mini responds in 200–500 ms, but complex queries take longer. We limit the context window and use few-shot examples.

Each of these problems is solved by choosing the right tool and tuning for the specific scenario. For example, in a telemedicine project we achieved p99 latency of 1.2 s by combining Silero VAD with local Whisper on GPU and streaming TTS. According to the official OpenAI Realtime API documentation, end-to-end latency does not exceed 800 ms when using server-side VAD.

How we build the Speech-to-Speech architecture

We build the architecture on streaming components to minimize buffering. The basic pipeline:

Microphone → VAD → STT → NLU/LLM → TTS → Speaker ↑ ↓ Endpointing First audio chunk (600–800ms) (<300ms after TTS start) 

Key insight: we start TTS after the first STT chunk, not after full transcription. This approach reduces overall latency by 20–30%.

Full pipeline on OpenAI

import asyncio from openai import AsyncOpenAI import sounddevice as sd import numpy as np client = AsyncOpenAI() class VoiceAssistant: def __init__(self): self.conversation_history = [] self.system_prompt = "You are a helpful voice assistant. Answer briefly, 1–3 sentences." async def listen_and_respond(self): # Record via VAD audio = await self.record_speech() # STT transcript = await client.audio.transcriptions.create( model="whisper-1", file=("audio.wav", audio, "audio/wav"), language="en" ) user_text = transcript.text print(f"User: {user_text}") # LLM self.conversation_history.append({"role": "user", "content": user_text}) response = await client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "system", "content": self.system_prompt}] + self.conversation_history, ) assistant_text = response.choices[0].message.content self.conversation_history.append({"role": "assistant", "content": assistant_text}) print(f"Assistant: {assistant_text}") # TTS streaming async with client.audio.speech.with_streaming_response.create( model="tts-1", voice="alloy", input=assistant_text, response_format="pcm", ) as tts_response: async for chunk in tts_response.iter_bytes(1024): # Play chunks as they arrive audio_data = np.frombuffer(chunk, dtype=np.int16) sd.play(audio_data.astype(np.float32) / 32768.0, samplerate=24000) sd.wait() 

OpenAI Realtime API (optimal for production)

import websockets async def realtime_voice_assistant(): url = "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview" headers = { "Authorization": f"Bearer {OPENAI_API_KEY}", "OpenAI-Beta": "realtime=v1" } async with websockets.connect(url, extra_headers=headers) as ws: # Configuration await ws.send(json.dumps({ "type": "session.update", "session": { "voice": "alloy", "instructions": "You are a voice assistant. Answer in Russian.", "turn_detection": {"type": "server_vad"} } })) # ...event handling 

How we reduce end-to-end latency

The critical factors are parallel processing and choosing the right endpointing algorithm. We use webrtcvad with aggressiveness 1 and dynamic timeout. In production with Realtime API, server-side VAD is 100–200 ms faster than client-side. Additionally, we cache embeddings for frequent commands (p99 latency drops by 15%). According to official OpenAI documentation, end-to-end latency does not exceed 800 ms. Savings on call center operators can reach 70%.

Which technologies do we use?

For local inference we use ONNX Runtime and vLLM — GPU utilization reaches 85%. Comparison: classic pipeline is 2–3 times slower than Realtime API, but gives more control over the voice. The cost of processing one minute of audio in the cloud is fractions of a cent.

Process of work

1. Analysis — measure current infrastructure, voice requirements, SLA (from 2 weeks).
2. Design — select components (OpenAI/local), design integration (from 3 days).
3. MVP implementation — basic chain VAD→STT→LLM→TTS with streaming (1 week).
4. Testing — A/B tests with users, measure p99 latency, adjust endpointing (3 days).
5. Deployment — set up CI/CD, monitoring in Grafana, alerts on latency (2 days).
6. Optimization — fine-tune Whisper for accents, LoRA for LLM, TTS voice for brand (optional).

What's included in the work

• Architecture and API documentation.
• Source code of the assistant with comments and tests.
• Integration with your CRM/telephony via REST.
• Team training (2–3 hour workshop).
• 1 month post-launch support with bug fix guarantee.

Estimated timeline

MVP voice assistant — from 1 week. Full production with Realtime API — 2–3 weeks. The cost is calculated individually based on integration complexity and customization scope. Get a consultation for your project — we'll evaluate it turnkey.

Performance metrics

OpenAI Realtime API: end-to-end latency ~500–800 ms.

Contact us to get a consultation and a detailed implementation plan. We guarantee: your voice assistant will respond faster than 1.5 seconds.