Why OpenAI Realtime API breaks in live conversations
On a demo, a voice assistant responds instantly, but on a live line the conversation falls apart: pauses, overlapping utterances, connection resets.
The reason is that OpenAI Realtime API maintains not a request-response scheme but a continuous two-way stream of audio and control events: a failure inside the stream breaks the entire dialogue, not just one utterance.
The first bottleneck is the response latency of the voice assistant. It does not come from one place: speech recognition during the call, waiting for the model, synthesis, and delivery to telephony add up into a chain, and the subscriber hears the total pause.
As soon as it exceeds the usual speech tempo, the person decides the connection is lost and starts speaking again — hence interruptions in the dialogue and broken logic.
Interruption is a separate task, not a side effect.
It is necessary to detect the start of the subscriber's speech, mute the echo from the assistant's own voice, and cut off the current generation mid-sentence; otherwise the bot talks over the client and loses the thread of conversation.
The second class of failures is connection drop on the line.
A long session does not survive a transition between networks, a carrier change, proxy, and mobile internet: the stream breaks, the conversation context goes with it, and the subscriber has to repeat everything from scratch.
This cannot be cured with settings — it requires state preservation and reconnection with history reload.
Next we break down how this is closed at the architecture level: where to place session boundaries, how to maintain speech tempo, and what to store in the log so that a failure can be reproduced from logs.
Effect of the voice layer: metrics before and after implementation
We take measurements on the same stream of requests: we label dialogues by channel, calculate the share closed without an operator, time to first response, and conversation duration until issue resolution. We compare before and after on identical scenarios — otherwise metrics prove nothing.
Speed. Previously a request went through the chain "phrase recording — recognition — response — speech synthesis," and the pause between utterances reached 5-8 seconds: the person either interrupted the robot or ended the call.
The voice layer on OpenAI Realtime API maintains stream exchange, response fits within 1-2 seconds, and the dialogue stops differing from a conversation with a live specialist.
Automation of request processing. Before implementation, the scenario almost always reached an operator — the auto-responder only clarified the topic.
After, the share of automatic dialogues stays in the range of 60-70% on typical scenarios: recording, transfer, request status, address clarification. Request processing time is reduced from 4-6 minutes to 1.5-2.
Peaks and failures. One service handles hundreds of simultaneous conversations, the "all operators busy" queue stops occurring, and the load on operators drops to non-typical and disputed cases.
A whole class of failures goes away: drop during a pause, misunderstanding of interruption, repeated recording of the same phrase. Which of these will work in your request stream will be shown by scenario analysis and measurements on your data.
Realtime API modes and when each is suitable
What we compare
OpenAI Realtime API works with voice in several modes, and the choice between them determines the solution architecture. A two-way voice channel keeps the session open: speech goes as a stream, the response comes as a stream, the interlocutor can interrupt.
Recognition of recorded audio is arranged differently — the file is processed entirely, the result appears after completion, but tasks are submitted in batches and repeated without losing state.
The boundaries between options are visible by three questions: is there counter-speech at the moment of conversation, is a reaction needed before the end of the phrase, and what to do with accumulated recordings.
| Option | When suitable | Limitations |
|---|---|---|
| Continuous two-way channel | Incoming calls, service booking, support — where a response is expected immediately | Requires a stable channel; if dropped, the session is raised anew; pauses consume resources |
| Recognition of recorded audio | Call archives, meetings, interviews, podcasts | No interactivity: response only after processing; noisy recordings require preprocessing |
| Hybrid processing scheme | Voice assistant on the line plus analytics on the same conversations | Two contours and a shared queue: observability and maintenance are more complex |
How we choose
Hybrid scheme aligns quality: live dialogue goes over the streaming channel, and the same conversations after completion are run in batches — for labeling, search through transcripts, and quality control.
The price is a coordinated session scheme and a shared task queue, so we fix the mode before work begins and check it on the customer's recordings.
How we implement the voice channel: stages of work
We start not with model configuration but with analysis of how the line lives now: which scenarios operators close, where calls are lost, what happens under overload. Without this, integration of OpenAI Realtime API repeats existing bottlenecks — just more quietly for monitoring.
- Voice channel audit. We collect stream statistics: share of typical requests, peak intervals, duration, causes of drops. From transcripts we compile a list of scenarios where the bot will replace an operator without quality loss.
- Dialogue flow scheme. We draw conversation branches and handoff points to a human, and coordinate them before code: editing the scheme takes hours, reworking the scenario takes days.
- Channel assembly. We connect telephony, configure recognition for the customer's vocabulary — service names, SKUs, names — and set interruption rules and silence pauses.
- Escalation regulations. We define thresholds for reaching an operator: repeated clarification, irritation in speech, response delay, non-working hours. Conversation context goes together with the call.
- Load testing. We run simultaneous calls above peak, look at first response latency, share of unrecognized utterances, and behavior during a drop. Based on results, we adjust limits for parallel sessions.
- Solution acceptance. We call through a checklist of scenarios, compare recordings with expected logic, and record metrics. We hand over access, documentation, and instructions to operators.
After handover, the channel remains observable: we review dialogue logs regularly and further train scenarios on real formulations.
What remains with you: configurations, schemes, regulations
A voice assistant on OpenAI Realtime API is not a single service but a set of settings and connections that are easily lost if they remain only with the contractor.
We hand over the work so that your team can change the assistant's behavior, connect new systems, and investigate incidents without us.
- Voice assistant configuration: session parameters, system instructions, callable functions, interruption thresholds, and timeouts — versioned files in your repository, not settings in someone else's panel.
- Integration scheme: flows between telephony, session server, model, and CRM, indicating where dialogue state lives and where events go after the call.
- Access policies: roles and visibility scopes, key rotation procedure, secret storage, and rules for releasing a new version to production.
- Operator instructions: how to take over a dialogue from the assistant, what is visible on the screen, how to mark unsuccessful scenarios, and where to report failures.
- Support regulations: frequency of dialogue quality checks, control metrics, threshold for revising configuration, and procedure for rollback to a previous version.
- Test contour: a collection of reproducible dialogues that is run before every settings change and catches regressions in recognition and logic.
The set is sufficient to hand over support to your team or another contractor without assistant downtime. We fix the composition of artifacts in the contract before start.
Case: support line before and after restructuring
Before restructuring, the support line relied on a voice menu with four levels and a queue to operators.
Average time to a live person response — 47 seconds, share of requests closed without an operator — 34%, and 12% of calls dropped at the order clarification stage: the client was silent, and the menu reset the session.
We moved the first level to a streaming voice assistant: the utterance is recognized and voiced in one channel, the pause between question and answer fits within 1.8 seconds — on OpenAI Realtime API this is a standard mode, not a "recognition — synthesis — response" patch-up.
Scenarios and routing rules are moved to a separate service, so edits are made without rebuilding the core.
After a month of operation, the metrics are: share of successful dialogues without an operator — 71%, line response time — 1.8 seconds, drops — 3%.
A whole class of failures is gone: silence longer than a few seconds and interruption no longer break the session — the assistant yields the initiative and returns to the conversation at the right step.
What remained under control
Every dialogue is recorded and labeled, disputed cases go to an operator together with the transcript, and switching to a live line works in one step — the person receives context rather than starting the conversation anew. Shift load is now planned based on actuals, not peak call share.
Technical doubts worth resolving before start
Doubts before start are usually engineering ones: will the line withstand real sound, will the assistant start interrupting, will the new logic match what is already working.
Below are five questions we close before estimation, even before building a session on OpenAI Realtime API.
Noise and echo on the line?
Echo suppression — the assistant does not hear itself — noise reduction, and automatic gain control. We calibrate the speech detection threshold on real customer calls, not studio recordings, so false triggers go away.
What happens during interruption?
The session catches the start of the interlocutor's speech and stops generation: the unfinished phrase is not played to the end, context is cut at the moment of interruption.
Otherwise the assistant speaks through the client and the conversation falls apart within a few utterances.
Transferring current dialogue logic?
Branches, checks, and data transfer to internal systems are described as states and mapped onto the new model. First we fix the current logic on a scheme — behavior remains familiar for both operators and clients.
Who answers — assistant or human?
The assistant closes typical scenarios itself, and on disputed topics hands over to an operator — with a transcript and brief summary, so the person continues the dialogue rather than starting over.
Conversation recordings?
Where audio is stored, how long it is retained, who has access, what is cut out during anonymization — we configure according to customer requirements. The assistant can also work without saving sound: this does not affect the dialogue.
Let's discuss your task: what to send for evaluation
To come back with a project estimate rather than a broad range "from and to," we need a minimum set of input data. The more specific the input, the fewer assumptions in the phased implementation plan — and the more accurate the solution scheme for your scenario.
- Dialogue scenario: which tasks the voice assistant closes itself, what it clarifies, and at what moment it hands over to a human.
- Channels: phone line, website widget, mobile app, or built-in microphone on a device.
- Speech profile: whether a continuous streaming dialogue on OpenAI Realtime API is needed or recognition with subsequent text processing is sufficient.
- Example utterances: 20-30 real customer phrases, including interruption, silence, noise, and language mixing within one phrase.
- Response requirements: pause before first sound, supported languages, share of requests that may be transferred to an operator.
- Systems for integration: customer records, knowledge base, booking schedule, billing — and in which direction data is transferred.
- Constraints: where conversation recordings are stored, what must not be sent to external services, which logs must be kept.
- Current contour: what already works, which telephony and interfaces are available, and who owns the system on your side.
Send what you have: a draft script, a call recording, a flow scheme. We will fill the gaps during analysis — discuss the task, fix the stages, and come back with an estimate, stage plan, and solution scheme.
How does Realtime API differ from processing recordings in parts?
The difference is not in speech recognition quality but in where the boundary between recording and response lies.
With sequential processing, the channel works in portions: a fragment is recorded, waits for the end of the utterance, goes for transcription, and only after that a response is formed.
While this chain is running, the interlocutor is silent — in a live dialogue this is a noticeable pause, and this mode does not support interrupting the speaker at all.
<cite>OpenAI Realtime API</cite> maintains a continuous two-way channel: recognition runs in parallel with the conversation, audio goes as a stream, the end of the utterance is determined from the sound itself, and the response begins without waiting for the fragment to close and be saved.
Interruption here is an event in the same channel: the response stops mid-word rather than being finished to the end. This is why clarifying questions, topic changes mid-phrase, and short confirmations like "yes, correct" work.
Sequential processing remains the right choice where there is no rush: analysis of already recorded conversations, call transcription, brief summaries based on communication results.
Here accuracy of wording and repeatability matter more than latency, and the result is reproducible — analysis can be rerun without asking the person again.
When choosing an operating mode, we look at three things: whether reaction during speech and interruption are needed; how stable the channel is — a continuous session must be recoverable and missing pieces resent; and what to do with the final text.
A combination is often used: the voice channel responds in real time, while final transcription runs sequentially — reports are built from it, quality is checked, and disputed dialogues are analyzed.