Smart Reply in the app: why the suggestions are irrelevant
Irrelevant suggestions are almost never a problem with the model itself.
Nine times out of ten, Smart Reply breaks at the input: the last message goes into the context without the conversation history, the language is detected from the first token, and the tone is not passed along at all.
The model honestly generates a reply to what it was given — and the result is polite nonsense that the user deletes and types themselves.
The second source is generation latency. A suggestion lives exactly as long as the user holds their finger over the keyboard: if the reply arrives after the person has started typing, it is no longer needed.
The race for quality through a large server-side model almost always loses on response time, so the workable scheme is a fast local candidate on the device plus server-side refinement that replaces the suggestion if it makes it in time.
The third layer is conversation context and tone. A support thread, a personal chat, and a work group require different vocabulary for the same meaning.
If the interlocutor's role, the history of recent messages, and the language of the conversation are not passed into the suggestions, reply relevance drops visibly: the user sees formal phrases where they expect a short "ok, got it."
The cost to the product is measurable: the share of accepted suggestions falls from its usual values to single-digit percentages, and irritation from irrelevant options carries over to the entire app.
So let's break down step by step exactly where the context is lost and how to restore meaning to the suggestions.
What an improved suggestion system delivers
An improved suggestion system is measured not by the "smartness" of the model, but by how often the user accepts it.
The acceptance rate is the baseline metric: with standard autocomplete it stays within 8-12%, and after tuning ranking for a specific messaging scenario it grows 2-3 times.
The growth comes not from a larger model, but from context: a suggestion takes into account whom the user is replying to, what was in previous messages, and in what tone they wrote earlier.
Response speed is the second metric, and here what matters is not only the latency of the suggestion itself, but also the time from opening the chat to the sent message.
When options appear immediately, without waiting for the network, the user spends noticeably less time on a reply, and the share of abandoned drafts drops.
We keep generation on-device for typical cases and go to the server only when a request requires a heavy model — that way the suggestion does not depend on connection quality.
Reduced manual input is directly tied to autocomplete quality: if suggestions are relevant, the user types less than half of the characters manually in short replies.
This is not just convenience — on weak devices and while traveling, battery and data are saved, and the interface stays responsive.
Resilience to failures is a mandatory condition, otherwise the gain is nullified. If the model is unavailable, the app serves pre-prepared options and does not block input; suggestion errors do not affect message sending.
We separately test behavior when the network is lost, when the context is empty, and when the language changes mid-conversation — the system degrades predictably rather than breaking the core scenario.
We capture these four metrics before and after the improvement: they show what exactly changed in the product, not just in the report.
Approaches to generation: comparing the options
The choice between a local model, server-side generation, and a hybrid scheme is not about "trendiness" but about three constraints: data privacy, response latency, and inference cost at scale.
A local model computes directly on the device and does not send text anywhere — but it runs up against the phone's memory and heat. A server-side model delivers the quality of a large model but adds a network round trip and costs on every request. A hybrid removes some of the trade-offs but requires routing logic.
| Option | When it fits | Limitations |
|---|---|---|
| Local model | Sensitive data (messaging, healthcare), offline scenarios, simple 3-7 word suggestions | Weaker on long context, noticeable battery drain, limited by model size on the device |
| Server-side model | Complex answers, long context, consistent quality across all devices | Dependence on the network, latency, inference costs grow linearly with the number of users |
| Hybrid scheme | A mass-market product with different scenarios: short ones on the device, complex ones in the cloud | Harder to test, requires switching and degradation logic when the network is lost |
In practice, the hybrid wins most often: short suggestions are generated on the device in fractions of a second and cost nothing, while heavy requests go to the cloud.
The key is to define the routing rule and the behavior when there is no network in advance, otherwise the user will get an empty screen instead of a suggestion.
Let's break down how this is handled for your scenario.
Stages of Smart Reply implementation in a product
Implementing Smart Reply is not "plugging in a ready-made module" but a sequence of engineering decisions, where each subsequent step relies on the data from the previous one.
Below is the route we follow on a project: from analyzing the current logic to handing over a working system with support regulations.
-
Audit of logic and data. We break down how text input, message sending, and history storage are currently structured: where suggestions are generated, whether there is access to the dialogue context, how attachments and multilingual messages are handled.
The output is a map of bottlenecks and an understanding of what data is missing for high-quality suggestions.
-
Processing scheme and privacy. We design the flow: what is computed on the device, what goes to the server, how content is anonymized, and how personal and group chats are separated.
We separately define behavior under weak network conditions and in offline mode — a suggestion must not block input.
-
Model configuration. We choose an approach for the task: built-in platform mechanisms for basic scenarios or fine-tuning on anonymized product data for industry-specific vocabulary.
We configure the length and number of suggestion options, the confidence threshold, and filters for unwanted content.
-
Quality validation. We assemble a set of test dialogues and metrics: the share of accepted suggestions, time to send, rejections, and edits. We run scenarios across different languages and devices, comparing behavior before and after.
-
System handover. We deliver the configured setup, the processing scheme, filtering rules, and instructions for updating the model. The client's team receives a clear regulation: how to change parameters, how to read metrics, and when to retrain.
Such a route removes the main risk — suggestions that annoy users instead of speeding up communication. Describe your task — we will come back with an estimate and a plan for your product.
What the customer's team retains
Work with the Smart Reply model does not end at release: the customer receives not a "black box" but a complete package their engineers can use to develop and maintain the feature independently. Below is exactly what we hand over.
- Inference configurations: generation parameters, confidence thresholds, suggestion filtering rules, and sentiment reference lists — with comments on what changes and under what conditions.
- Data flow diagrams: from user input through preprocessing and the model to variant ranking and logging — indicating where texts are stored and for how long.
- Access policies: role separation for data exports, anonymization before training, logging of model requests, and incident response rules.
- Integration documentation: client library contracts, offline mode handling, behavior when the service is unavailable, and degradation to a local model.
- Update procedures: how a new model version is rolled out, how suggestion quality is validated on a reference sample, and how to roll back to the previous one without losing history.
- Quality metrics: a set of checks for suggestion accuracy and relevance, the share of accepted variants, and the rules by which degradation is recorded.
- Support instructions: typical failures, log-based diagnostics, and the escalation procedure to our team.
Each artifact is accompanied by a short breakdown — why this particular decision was chosen and what will happen if it is changed.
This way the customer's team is left not with a set of files but with a working picture of the system: it is clear where to edit the configuration, where to update the access policy, and which step of the procedure triggers retraining. Describe your task — we will come back with an estimate and a handover plan.
Case study: how the share of accepted suggestions grew
What it was before
The app is a messenger with text chats and attachments. Reply suggestions lived on three hardcoded phrases like "Ok, got it" and "Thanks!", identical for all conversations.
The share of accepted suggestions hovered around 4%: users either didn't notice the row or dismissed it because the suggestion didn't match the context of the conversation.
Every rejection meant a return to manual input, and in long conversations it also meant a loss of pace: the person had already formulated the thought, but had to type it from scratch.
What was changed
We assembled the conversation context — the last messages from both sides, the language, the tone, the presence of a question at the end. A fast on-device model was kept for short replies, while heavy scenarios went to server-side generation with caching of frequent phrasings.
Suggestions began to appear the moment the user opens the keyboard, rather than after tapping the input field. We added feedback collection: accepted, rejected, edited — and retraining once every two weeks on these signals.
What it became
The share of accepted suggestions grew from 4% to 23% — nearly sixfold. The average reply time in a conversation dropped by a third, and the share of messages sent without edits after a suggestion reached 61%.
The load on manual input fell most noticeably in scenarios with short business replies: there the suggestion covers the entire response.
We worked on the current branch of ML Kit for on-device text parsing and on server-side generation for complex cases — this struck a balance between response speed and the quality of phrasings.
How does on-device processing differ from server-side processing?
On-device processing keeps the model and inference inside the client: the message text never leaves the phone, and the response is generated locally. Server-side processing sends the context to a backend, where a larger model runs. This is where all four trade-offs diverge — privacy, latency, compute cost, and generation quality.
Privacy is solved architecturally, not through promises. With local inference, the conversation physically never leaves the device — this eliminates an entire class of risks when handling sensitive data and simplifies passing app store reviews.
On the server, the same result is achieved only through explicit measures: channel encryption, refusal to store the source text, and tenant isolation. Each of these measures must be designed and verified.
Latency in local generation does not depend on the network: the response appears within tens of milliseconds and works in the subway, on a plane, or while roaming.
Server-side processing adds a network round trip and a queue on the backend — on a stable connection this goes unnoticed, but on a weak one the connection drops and the suggestion arrives late.
For a scenario where the user is typing and waiting for a reply option right now, this is a decisive difference.
Compute cost is structured as a mirror image. Local inference is not billed per request, but it consumes the device's battery and memory — the model has to be quantized and limited in size.
The server pays for every call, but it is not constrained by the phone's resources and scales easily under peak load.
Generation quality is the price of the trade-off. A compact on-device model produces short, relevant suggestions but falls short on long context, rare languages, and nuanced style.
A server-side model is more accurate and better maintains the tone of the conversation, but it requires sending data outside.
In practice, a hybrid approach is used: a draft version is computed locally for an instant response, and the server is brought in where accuracy is needed and the user has given consent.
Common technical concerns
Questions about Smart Reply almost always come down to four things: languages, offline mode, model updates, and response quality control. Below is how this works in practice.
How does the app support multiple languages, including Russian?
The baseline scenario is a local on-device model: it produces draft responses in dozens of languages without contacting a server.
If you need wording tailored to a specific market and industry vocabulary, we add fine-tuning on the customer's conversation corpus — separately for each language.
We handle mixed-language dialogues (Russian + English in the same thread) by detecting language at the message level rather than for the entire chat: otherwise suggestions end up in the wrong language.
What happens when the user has no network connection?
Local inference works without a network — suggestions appear as the user types.
Server-side scenarios (heavy model, fresh data) are disabled offline, and the interface does not show an empty block: the app serves templates from the local cache or hides suggestions until connectivity is restored. This way the user does not see "stuck" elements.
How are models updated without an app release?
We move the model and dictionaries into a separate layer that is updated independently of the build. This makes it possible to change wording and add languages without republishing to the stores.
The update rolls out in stages: first to a small share of users, then to everyone — if the metrics have not dropped.
Who is responsible for the quality and safety of suggestions?
We monitor quality with metrics: the share of accepted suggestions, time to first variant, and the share of manually rejected ones. We filter responses for toxicity and personal data leaks — the model must not suggest wording that the user is not ready to send. On every release we run a set of test dialogues to catch regressions before rollout.
If you still have concerns about your product — describe the scenario, and we will get back to you with an assessment and a plan.
Let's discuss your task and come back with a plan
To come back with an estimate and a work plan rather than generalities, we need to understand your Smart Reply scenario in full: where the suggestions appear, in which language, and at what volume. Describe the task — we will ask clarifying questions and send a plan for the next steps.
What we will clarify in the first conversation:
- Where exactly the suggestions are needed: replies in a messenger, comments, support, reviews — the generation logic depends on this.
- Languages and tone: one language or several, whether the writing style of a specific user needs to be taken into account.
- Where the computation happens: on-device, on a server, or hybrid — this affects privacy, speed, and operating cost.
- Data for training and tuning: whether there is a history of conversations, dictionaries, restrictions on sensitive topics.
- Constraints: confidentiality requirements, offline mode, integration with the existing backend and authentication systems.
- Success metrics: share of accepted suggestions, response speed, user coverage.
As a result, you will receive a work plan with stages, an estimate of timelines and team composition, an architecture diagram for your scenario, and a list of risks with options for mitigating them.
If you already have a prototype or prior work — we will look at what can be reused and what needs to be rewritten.
Describe the task in free form: what the product is, who the audience is, what result is needed. We will come back with an estimate and a plan.