Voice assistant in a mobile app: where it breaks
A voice assistant in a mobile app breaks not during demo recording but in a live scenario: in a car, on the street, in a shopping mall.
Where the speech interface must work, speech recognition in the app receives not a clean signal but a mixture of voice, music, and other people's conversations — and model confidence drops precisely on the meaningful words: address, amount, item name.
Voice input failures are rarely isolated.
Background noise masks a key word, the user repeats louder — the assistant hears something else; the connection drops in an elevator or between stations, and the request goes nowhere without a clear response; a long pause for thought closes the session, and the person returns to the first screen, not understanding whether they were heard or not.
A separate category is recognition errors on ambiguous wording.
"Two" and "twenty," "for two days" and "for two," a street dictated letter by letter — the assistant either guesses or asks again in a loop, and trust in voice input falls faster than in manual input.
The most costly thing is loss of context between steps: you dictated an address, size, time, but the session state was not saved. Then every pause erases previous answers, and the order has to be started over. We explain how this is addressed.
What changes after implementation: speech as the primary interface
Input stops being a form. The voice layer parses the phrase into intent and parameters and passes them to the same operations that buttons trigger — it is an add-on over the app's existing logic, not a second product with its own business logic.
The first measurable effect is the share of scenarios resolved without a human agent. In telemetry it is counted directly: the session ended with an action, not a transfer to a person.
On repetitive operations — check status, repeat an order, book an appointment — the metric reaches 60–80% with carefully collected prompts.
The second is the speed of repetitive actions. A screen with four or five steps collapses into one phrase, and order placement is reduced from two or three minutes to tens of seconds.
The savings accumulate specifically in high-frequency scenarios: a person returns to them many times, not once when first getting acquainted with the app.
The third is a whole class of input failures: typos, incorrect format of a number and date, a missed required field.
Voice input returns already parsed values, and before an irreversible step such as a charge or cancellation, the app reads back the parameters and asks for confirmation.
The fourth is offloading the first line: typical questions go to self-service, while nonstandard cases remain with agents.
Recognition works in streaming mode — Yandex SpeechKit returns intermediate hypotheses before the end of the phrase, so the response starts earlier, and interruptions are handled correctly.
All four metrics are captured in the production environment, and they show where the layer works and where it needs refinement.
Speech recognition scenarios: comparison of operating modes
The recognition mode is chosen not by preference but by scenario: a short command "set a timer" and a ten-minute note dictation place different demands on speech processing.
That is why several assistant operating modes usually coexist in one app, and they switch imperceptibly to the user.
On-device speech recognition keeps audio on the device: it works in the subway and on a plane, does not require a connection, and does not send voice outside.
Limitations are a small vocabulary, weaker quality on noisy and mixed speech, and the model takes up space in the app bundle.
Cloud speech services provide accuracy on complex recordings, a broad vocabulary, and model updates without a release, but they depend on connectivity and involve sending voice to a server.
| Option | When it fits | Limitations |
|---|---|---|
| On-device | Short commands, offline scenarios, privacy-sensitive data | Narrow vocabulary, weaker in noise, larger build size |
| In the cloud | Dictation, long phrases, multiple languages | Stable connection required, voice goes to the server, response depends on the network |
| Hybrid | Mixed scenarios, unstable network | More complex switching logic and testing of both paths |
Hybrid processing is most often the answer to choosing a recognition scheme: commands go to the local model, dictation to the cloud, and when connectivity is lost the assistant finishes on the device and queues the transmission.
The boundary between modes is set by scenario type, phrase length, and network state — at the design stage, not after release.
How implementation proceeds: from scenario audit to handover
The workflow is structured so that the assistant learns from real requests, not invented phrases.
First we look at what people already write and say to support, and only then design dialogues — otherwise the first nonstandard wording breaks the scenario and the conversation goes to an agent.
- Dialogue scenario audit. We export the request history, classify it by topic, and calculate what share the assistant resolves and what will have to go to a human. The output is a list of scenarios prioritized by frequency.
- Annotation of real dialogues. We collect a reference sample of live utterances: how people ask, how they ask again, where they drop off. We use it to tune recognition and intent parsing, and record disputed cases in a separate set for checks.
- Dialogue design. We define branches, clarifying questions, fallback scenarios, and escalation rules to an agent. Critical actions — payment, changing data — only with explicit confirmation.
- Assistant setup. We connect speech recognition and response synthesis, voice profiles, confidence thresholds, integration with the accounting system and knowledge base.
- User testing. We launch on a limited share of traffic: we look at the share of conversations completed without an agent, the number of clarification requests, and the reasons for failures. We make edits based on actual logs, not impressions.
- Error closure. We analyze every unsuccessful branch, retrain on new wording, and run regression on the reference sample so as not to break what already works.
- Handover to the team. We hand over configurations, the dialogue scheme, the phrase editing policy, and monitoring by key metrics — after that the client runs the assistant on their own.
What the client retains after assistant launch
An assistant is not a microphone button but a set of engineering decisions: how speech is recognized, how intent is extracted from a phrase, what the app does when the network drops. If these decisions remain in chat threads, any scenario edit turns into archaeology.
We hand over the project so that the client's team can develop the assistant themselves. The set of artifacts in hand:
- Dialogue scheme — a tree of scenarios with branching points: what counts as a command, where the assistant clarifies, where it transfers the request to an agent.
- Error and retry handling rules — how many times to ask again when recognition is uncertain, when to offer text input, when to return a step back.
- Voice data access policies — where recordings are stored, how long they live, who has access to them, and how they are deleted on request.
- Assistant documentation — a list of intents, supported wording, and instructions for adding a command without modifying the core.
- Support policy — which events count as a failure, where notifications arrive, how quickly an incident is handled, and how a release is rolled back.
- Repository and project handover procedure — code, configurations, access, and a description of the test and production environments.
- Development procedure — how to add a scenario or language without rewriting the core.
The handover is not an archive: we go through each item with the client's team until no gray areas remain. After that, our help is needed only for new scenarios.
Case study: how an assistant in a delivery app relieved support load
A food delivery chain with an app on iOS and Android: the peak load is Friday evening, and support is overwhelmed by identical questions: "where is the order," "repeat the last one," "change the address." Before the assistant was implemented, 12% of requests were resolved without an agent, and the rest went to chat and the line.
The scenarios were assembled not "for everything" but for three frequent intents: order status, repeat the previous cart, change address.
The assistant opens directly from the order card and receives session context — number, address, history — so it does not ask again for what the app already knows.
Recognition is built on a cloud service on the level of Yandex SpeechKit with intermediate results: the person sees the text as they speak, not after a pause.
Metrics before and after
| Metric | Before | After |
|---|---|---|
| Requests resolved by voice without an agent | 12% | 41% |
| Repeat order placement | about 3 minutes | 50 seconds |
| Dialogue breakdowns | every fifth | one in twenty |
| Recognition failures | 18% | 6% |
Dialogue breakdowns were reduced not by the model but by an escalation rule: after two unsuccessful clarification attempts, the conversation seamlessly transfers to an agent along with the transcript.
Recognition failures dropped due to short prompts about what exactly can be said, and replaying the phrase at a slower pace.
The number of requests to agents decreased by a third, and during peak hours the line was unloaded so much that the waiting queue stopped growing by the end of the evening.
How does on-device speech processing differ from cloud processing?
Short answer: the choice comes down to where the acoustic and language models are located — on the device or on a server. With local processing, speech is recognized directly in the app: open models such as Whisper run next to the microphone, without sending sound outside.
Cloud processing sends the stream to servers, where recognition quality is higher, and scenarios are almost unlimited by vocabulary.
Privacy of voice data differs fundamentally: locally, raw sound does not leave the device; only recognized text goes outside.
In the cloud, audio is transmitted over the channel and stored for some time, so encryption, a retention period policy, and explicit user consent are required. Offline, the local model responds in airplane mode; the cloud one does not work under the same conditions.
In practice, the choice of assistant architecture is rarely binary. We determine which commands must work without a network and which sound cannot be taken outside the device — they go to the local environment, while free-form wording and complex dialogues remain in the cloud.
The hybrid scheme provides fast response on frequent requests and removes failures related to loss of connectivity.
Common technical doubts before starting a project
Questions about accuracy, languages, and requirements for the app itself are resolved at the design stage, not after launch.
Below are the things most often checked before starting: recognition, domain logic, and dialogue scenarios are tested together, on real recordings, not separately.
How accurately does the assistant recognize speech in noise?
Accuracy depends more on recording conditions than on the chosen model: street, room echo, quiet voice near the microphone. We enable noise suppression and set a confidence threshold — when confidence is low, the assistant asks again or offers options rather than guessing. Tuning is done on recordings from real scenarios, not studio examples.
What about accents and mixed speech?
We build a domain vocabulary: terms, names, abbreviations — with pronunciation variants. Mixed phrases are parsed at the language model level, where some words are recognized in one language and some in another.
Disputed recognitions are checked by an engineer against an annotated sample, and the vocabulary is refined.
What does the assistant do if it does not understand?
We design the misunderstanding scenario explicitly: a clarifying question, narrowing the options, switching to manual input. Silence or a random guess is more irritating than a request to repeat. Every such case goes into a log — it is a ready-made list of scenario and vocabulary edits.
How are new languages added?
We make the architecture without hard language coupling: recognition, domain logic, and responses are separated. A new language means connecting a language model and translating scenarios, not rewriting the app. We start with one or two, and add the rest as demand grows.
What are the requirements for the app?
Microphone permissions, background operation, and predictable behavior on a weak network are required.
We fix the minimum supported operating system version so that recognition does not degrade on older devices, and we provide graceful quality reduction: the phrase is saved and goes for processing when connectivity appears.
Let's discuss the task: we'll review scenarios and come back with a plan
Project estimation starts not with numbers but with inputs: which scenarios the user talks to the app in, and what limitations your infrastructure has.
We will retrieve some data ourselves from the code and mockups, but a few things only you know — they are worth bringing to the task discussion.
- voice scenarios: what is handled by speech and what remains in the interface;
- platforms and minimum supported OS versions;
- whether speech recognition and synthesis already exist, or we select an engine from scratch;
- whether offline operation is needed, and what reserve of commands to keep offline;
- access: microphone, platform accounts, personal data processing rules;
- languages, industry vocabulary, and requirements for recognition accuracy;
- deadlines and the point by which a fully working scenario is needed.
We will review these inputs on a call and come back with a scope estimate, a phased implementation plan, and a breakdown of work — with a clear division of what we do and what is required from your side.
Describe the task in a convenient form: in text, a voice message, or a link to the current app — and we will suggest a time for a substantive conversation.