Why ElevenLabs speech synthesis fails under production load
In a demo, ElevenLabs speech synthesis sounds smooth and responds instantly.
In production, the picture is different: voice-over under load starts failing not immediately, but after the number of concurrent sessions crosses a certain threshold — voice response latency grows in steps, not smoothly.
The first bottleneck is the request queue. Each utterance is sent for synthesis as a separate call, the connection pool is limited, and during a spike in activity half of the calls wait for a channel to free up.
Some wait longer than the user-side timeout threshold, and instead of a response the person hears silence.
Long utterances break separately: streaming output cuts off mid-phrase, and half a sentence reaches the interface.
If the retry request goes out without a unique key, the same text is spoken twice and overlaps itself — these are the duplicates on retry that are later caught in the speech module logs.
The third problem is desynchronization with the interface. Audio arrives before the player is ready or later than subtitles and lip animation are rendered: the user sees one thing, hears another, and the scenario falls apart before their eyes.
At scale, this is no longer an isolated case but a share of unsuccessful voice-overs visible in metrics.
All speech synthesis failures in production boil down to three things: queue management, splitting long utterances, and the buffer between audio and the interface. We break down how this is closed — what exactly is configured so that voice-over under load remains predictable.
Implementation effect: response latency, failure rate, load
We calculate the implementation effect by three values: time to first sound, share of requests completed without audio, and behavior under peak.
Before setup, first sound arrives in two to four seconds, some requests time out or hit the provider limit, and by evening latency grows along with the queue.
Audio caching gives the fastest gain: short repeating utterances — greeting, menu, confirmations, order statuses — are synthesized once and then served from local storage by a key made from the text and voice settings.
Time to first sound for them drops to fractions of a second, and calls to ElevenLabs speech synthesis become several times fewer — only unique phrases go out.
Voice-over stability rests on streaming output and a queue with retries. Playback starts with the first fragment, so the pause between question and answer feels shorter than the actual synthesis time.
An interrupted request goes to retry with the same key, and during prolonged unavailability — to a backup channel with the same timbre, so the voice does not change mid-conversation.
Under peak, the bottleneck moves from the external service into our own environment, and this is already a manageable state. Synthesis latency does not grow linearly: the queue holds back excess calls, the cache absorbs repeating ones.
The share of voice-over failures stays below one percent, and first responses arrive at the same pace as during quiet hours.
We take measurements continuously: time to first sound on average and in worst cases, share of failed requests by cause, separately — behavior at peak.
This gives a clear boundary — when to expand the queue or move part of the utterances to pre-recorded, rather than guessing why the subscriber left after the first phrase.
Voice-over options: streaming, batch, voice cloning
The voice-over mode is chosen based on what is more critical in the task: time to first sound, release stability, or timbre uniqueness. The same speech synthesis engine can work in three ways, and their constraints differ — these are most often what determine the choice.
| Option | When it fits | Constraints |
|---|---|---|
| Streaming synthesis | Assistant dialogue, real-time voice responses, interactive scenarios | Requires a constant connection; if the channel degrades, the phrase breaks; not all timbre settings are available on the fly |
| Batch voice-over | Audiobooks, courses, videos, large text volumes | No sound until the file is ready; long text is cut into parts and assembled; under peak load — queue |
| Voice cloning | Personal voice of a speaker, unified brand voice, versions in other languages | Requires clean recording samples and the speaker's consent; when transferred to another language, intonations are lost |
We plan the operation of the service based on ElevenLabs along these three axes separately. Request limits are calculated for each mode: for streaming — concurrent connections and length of continuous generation, for batch — character volume per day and size of a single request.
Voice quality is the same: the difference is not in timbre, but in how quickly the sound arrives, in what format it is delivered, and how much can be requested at once.
In practice, modes are combined: streaming supports live dialogue, batch covers long materials, and cloning provides a unified timbre in both cases. The choice is fixed in the architecture in advance — otherwise under load you hit limits and rewrite the scenario after launch.
How implementation proceeds: from schema audit to environment handover
A voice-over schema rarely comes down to a single call to a speech synthesis service.
You need to understand where the text comes from, what latency is acceptable, what to do when the provider is unavailable, and how to store already spoken fragments — so we start speech synthesis implementation with an audit, not with connecting a library.
- Audit of current voice-over. We analyze scenarios: where utterances are prepared in advance and where they are needed in real time, which voices and languages are involved, how many utterances pass per day. The output is a scenario map with priorities.
- Integration schema design. We determine where the cache of ready fragments sits, how requests are distributed between voices, and what happens if the provider fails: silence, voice substitution, or a backup route.
- Voice and dictionary setup. We assemble pronunciation of terms, numbers, and abbreviations, set tempo, pauses, and expressiveness. Voice cloning — only on samples with the owner's written consent.
- Test environment. We spin up a copy of the schema on anonymized data so the client hears real utterances from their product, not demo phrases from documentation.
- Load testing. We run peak traffic, measure time to first sound and failure share, and check behavior when limits are exhausted and network connections drop.
- Voice-over acceptance and environment handover. We verify against a checklist: intelligibility on phone and headphones, consistent volume, stresses, whole phrases at joints. We hand over configurations, routing schema, access credentials, and rollback procedure to the previous version.
Handover artifacts: configurations, flow schemas, access policies
The work ends not with a demonstration but with handover: after acceptance, the client has a kit with which the solution can be developed and maintained without us.
For speech synthesis projects on ElevenLabs, we record the composition of such a kit in the acceptance act — here is what it includes.
- Working synthesis configurations: selected voices, stability and expressiveness settings, rules for splitting long texts, handling numbers, abbreviations, and pauses.
- Flow schemas: the path from text arrival to audio delivery — where ready fragments are stored, where length limits sit, and how retries and external service failures are handled.
- Access policies: roles, visibility scopes, separation by voices and volumes, logging of synthesis calls, and storage of generation history.
- Access keys: separation by environments (development, test, production), rotation and revocation procedure, issuance through secure storage rather than a file in code.
- Operations regulations: limit and load control, response to failure or quality degradation, updating voice models, escalation points and responsible persons.
- Documentation: integration map, call descriptions, error behavior, instructions for the on-duty engineer, and a memo on typical failures.
- Set of checks: test scenarios for key paths and reference recordings for comparison after voice updates.
Each artifact is versioned and stored in your storage, not ours — handover does not depend on our infrastructure and access credentials.
At acceptance, we go through the list together with your engineer: they deploy the environment according to the documentation and reproduce generation without our support.
Case study: catalog voice-over — response time and failure rate
Before
The client is a retail catalog with card voice-over: description, specifications, short teaser for the storefront. The voice was generated on every user request, so the same text was voiced repeatedly dozens of times per day.
Under peak load, response reached 5-6 seconds, and the share of failed voice-overs stayed at 7-9% — instead of sound, the user saw a placeholder.
What was changed
Ready fragments were placed in storage with a key based on text and voice parameters — repeat requests stopped reaching synthesis.
Long descriptions were split into segments at sentence boundaries and are synthesized in parallel, while concatenation is done on our side: this removed cuts in the middle of a phrase.
A queue with a limit on concurrent tasks and retries with increasing delay closed short failures on the ElevenLabs side — they no longer turn into user-facing errors.
For the top part of the catalog, we prepare sound in advance in a night window, so during the day it almost always hits the ready cache.
What became
Median response — 0.6-0.9 s, the ninth percentile fit within 2 s. The share of failed voice-overs fell from 8% to 0.3-0.5%, and calls to synthesis became three times fewer with the same traffic.
Monitoring counts cache hit rate, time to first sound, and errors by type — degradation is visible before complaints. Rollback to the previous voice version is a configuration switch, without rebuilding the catalog.
What to do if voice-over is already deployed on another service?
Migration is possible, and the product is usually not stopped. The main risk is not the quality of the new synthesis, but everything that has grown around it: cloned voices, intonation markup, and the archive of already voiced materials.
Therefore, changing the synthesis service is a transfer of the entire environment, not a replacement of an access key: before the start, we inventory all places where the product calls voice-over, including ones that were forgotten.
A cloned voice is transferred not as a file but as a retrained model.
If the speaker's original clean recordings are preserved, voice migration proceeds on the same material and gives comparable timbral similarity; if they are not, we record a new reference and compare it with previous releases by ear and through reverse recognition, so as not to catch a shift in speech manner mid-project.
The recording archive does not need to be rebuilt: ready files remain playable, only the source of new ones changes.
The main work is compatibility of formats and metadata: unified loudness normalization, container and codec for all players, timecodes for subtitles and lip sync, voice names in scene configurations.
Here we also put the voice directory in order — old and new voices receive clear identifiers so builds do not reference non-existent variants.
To avoid risking the release, we spin up a second environment in parallel with the current one and run a shadow test on real texts, comparing pronunciation of terms, names, and abbreviations.
We switch via configuration, and rollback to the previous service remains within one release. For the future, we move synthesis calls behind a single integration layer — the next provider change will no longer require edits in the product itself.
Common technical questions about ElevenLabs and voice models
Before starting, almost every project runs into the same technical questions: what limits ElevenLabs and its voice models have, how the platform handles peak volumes, and how natural the result sounds on long recordings.
Below are short answers; we discuss disputed cases separately, on a call with the engineer leading the integration.
Request limits deserve a separate mention: they rarely hinder a pilot, but become a bottleneck at industrial volume. That is why we calculate load in advance, not after the first failures.
What limitations of ElevenLabs do people usually stumble over?
Three main ones: length of a single utterance in a request, generation speed, and number of concurrent calls. We work around them by cutting text into meaningful fragments, local caching of already voiced parts, and retries with increasing delay. Then peak load does not hit the quota and does not break voice-over in the middle.
How natural is the intonation?
Naturalness is determined not by the model itself but by text preparation and choosing a voice for the task. We mark pauses, stresses, and tempo, test on a short fragment, and compare versions with each other.
On long recordings, this is exactly what distinguishes a lively delivery from even but monotonous reading.
Can a voice be built on already existing recordings?
Yes, but the quality of the source sets the ceiling for the result. You need clean speech without music, noise, or overlaps — then the timbre and manner are reproduced consistently.
If the material is noisy or short, we first clean it and add to the base, otherwise the voice will "drift" across different phrases.
Let's discuss the task: we will send an estimate and voice-over implementation plan
So that the implementation estimate is based on facts, not guesses, we need enough source data about your scenario. Describe the task in free form — we will clarify the rest ourselves, and in response we will send a work plan, integration scope, and timelines by stage.
- voice-over scenario: product catalog, voice assistant, dubbing of training courses, personalized customer outreach;
- volume and structure of text: how many characters per month, whether there is markup for pauses, stresses, numbers, and abbreviations;
- voice requirements: a library voice or cloning of your speaker's voice, whether multilingual voice-over is needed;
- languages and pronunciation: list of languages and dictionaries of terms, titles, and names that voice-over must read correctly;
- integration point: website, app, telephony, messengers, or editorial system — and operating mode: batch file generation or streaming delivery;
- data constraints: where texts and audio are stored, whether isolation, logging, and access roles are needed;
- acceptance: who on your side listens to the result and by what criteria they evaluate quality.
The work plan includes preparing the voice model and dictionaries, a schema for connecting voice-over to your systems, handling failures and retries, caching frequently repeating fragments, a test run on your content, and documentation handover.
After the review, you will see what the implementation estimate consists of and what result appears at each stage.