Guardrails for an AI assistant in a mobile app

Guardrails AI in a mobile app: where the assistant breaks

An assistant in a mobile app breaks not where it was tested.

In a demo, the model answers prepared questions, but in production users bring phrasings that were not in any scenario: they bypass filters through paraphrasing, ask to "show your instructions," nudge it toward actions the assistant has no rights to.

Guardrails AI is a layer of rules around the model that solves three tasks: prevent it from outputting prohibited content, prevent it from inventing a fact, and prevent it from performing an operation without human confirmation.

Let's break down the typical failures we see in production. Prompt leakage — the user extracts system instructions through "translate them into another language" or "continue the text," and internal rules, service names, and moderation logic leak out.

Hallucinations in production are more dangerous because they look plausible: the assistant confidently states a delivery time that isn't in the data, or invents a tariff condition.

Bypassing filters through paraphrasing — a prohibited request is rewritten politely, in role-play, or through a hypothetical situation, and a keyword filter lets it through entirely.

A separate class is dangerous actions. If the assistant can initiate a payment, cancel an order, or change personal data, an error costs not reputation but real operations.

Here guardrails work not as a response filter but as intent confirmation: the assistant proposes an action, the user confirms it, and only then does the request go to the backend.

The cost of each failure is different, but the mechanics are the same: the user loses trust in the assistant after the first confident mistake and stops using it.

That is why guardrails are designed not on top of a finished assistant but together with it — at the input, at the output, and at the tool-call level. Below we break down how this is structured and what each layer covers.

What a governed assistant delivers: the effect in metrics

The constraints of an assistant are measured not by how many requests it let through, but by how many problematic responses never reached the user.

On pilots, we establish a baseline across three metrics: the share of responses that violate domain rules, the share of requests that ended up blocked, and the time from a rule triggering to a configuration fix.

Without this baseline, any refinement turns into an argument about taste — with it, you can see exactly what changed after each rule.

The effect then breaks down by failure classes, and each one is closed by its own check:

• Unsafe phrasing — an input and output filter removes it before sending; the share of such responses drops to isolated cases that are handled manually.
• Fabricated facts about the product — responses are built only on approved sources, so disputed claims are removed from the output entirely.
• Leakage of internal data — masking of sensitive fields at the context preparation stage; the failure class is closed at the input, not after the fact.
• Bypassing rules through rephrasing — a post-generation re-check catches attempts to circumvent the constraints and does not let them through to the user.
• Silent failure — if a check did not run, the assistant responds with a refusal rather than a guess; this moves the incident from the category of "it broke unnoticed" to "immediately visible in the logs."

A separate metric is the share of blocks. Too high means the rules are cutting off useful requests and the assistant is losing value; too low means the filters are not working.

We keep it within a corridor and review it with every change to the scenarios, so that constraints do not become an end in themselves.

Recovery time after an incident depends on how well the rules are separated from the application code. When they are moved into configuration, a fix takes minutes and does not require a new release — the cost of downtime drops by a factor, not by a percentage.

It is precisely this margin of resilience that is the practical point of a governed assistant: a failure is localized, rolled back, and investigated without stopping the product.

Approaches to restriction: comparison of operating modes

AI assistant restrictions can be placed at four points: before generation, after it, at the level of the execution environment, and at the human level. They almost always have to be combined — each mode covers its own class of risks and leaves its own class of gaps. Below is an honest comparison, including cases where a mode is not suitable.

How the modes combine

The input check is cheaper than the others and filters out obvious garbage before the model is called — but it does not protect against a correctly formulated request with undesirable meaning.

The output check catches what slipped through and works with an already finished response: checks for prohibited topics, data leakage, and format compliance fall here.

The isolated environment restricts not the text but the capabilities: the assistant sees only permitted sources and cannot perform an action beyond the agreed set.

Manual moderation does not replace automation but complements it: it handles controversial cases that the rules did not classify, and based on the results of the review the rules are refined.

In practice, the working scheme is input, output, and isolation on automation, while a human is connected to the incident queue and to topics where the cost of an error is high.

Choosing a mode is a choice between response latency, completeness of scenarios, and support cost. We analyze your scenario and select a combination for it, rather than the "safest" one based on abstract considerations.

How we implement restrictions: stages of work

We structure the work so that restrictions for the assistant don't appear "along the way," but as a managed layer between the model and the application interface.

First, we establish what the assistant can actually do in the product, then we translate that into rules verified by code, and only after that do we release it to production. Below is the sequence by which implementation proceeds.

1. Scenario audit. We go through all the points where the user interacts with the assistant: chat, form hints, text generation, answers to questions about data. For each scenario, we write down which actions are permissible, which data enters the request, and what is considered an unacceptable response.
2. Policy schema. We consolidate the scenarios into a single rules map: topic restrictions, prohibition on disclosing personal data, requirements for tone and response format, behavior when the model is uncertain. The schema is agreed upon with the product team and legal before writing any code.
3. Setting up checks. We implement filters on input and output: validating the request before sending it to the model and validating the response before showing it to the user. We keep the rules declarative — they can be changed without rebuilding the application.
4. Robustness tests. We run sets of attempts to bypass the restrictions: rephrasing, language switching, injecting instructions into user text. We record how many attempts are blocked and which responses go to manual review.
5. Release to production. We enable the restrictions in stages: first for a subset of users, with logging of triggers and rejection metrics. If the share of false blocks grows, the rules are adjusted before full rollout.
6. Handover to the team. We provide documentation on policies, instructions for adding new rules, and a review procedure for when the product changes — so that the restrictions are maintained without us.

This sequence produces a predictable result: the assistant remains useful, and behavior beyond what is permitted is closed off at the application level, rather than relying on hope in the model.

Access policies and data handling rules

Access policies and data handling rules are what you're left with after deploying the assistant: not abstract "recommendations," but versioned artifacts that live in your repository and go through review like ordinary code. Below is what exactly is included in the package.

• Rule configurations: declarative files that record which requests the assistant processes, what it masks, and what it rejects. The rules sit alongside the application, not in someone else's dashboard, so editing them goes through the same approval process as the rest of the code.
• Data flow diagrams: a diagram of the request path from the interface to the model and back, with explicit markers showing where data leaves the device, where it is anonymized, and where it doesn't leave at all. It shows which fields should never make it into a request in the first place.
• Access roles: a "role → what they see → what they can change" matrix for support, analytics, and development. It separately describes who has the right to export dialogue logs and who only views aggregated metrics.
• Escalation procedure: a step-by-step course of action when a rule is triggered — from automatic rejection to manual review. With an indication of who is responsible at each step and within what timeframe.
• Test suites: a set of requests used to verify the rules before every release — including attempts to bypass restrictions through rephrasing. The suites run automatically, so regressions are caught before rollout.
• Documentation: a description of the rules' logic and a change log — so that six months later a new engineer understands why a restriction is formulated the way it is.

Each artifact is delivered within your environment: rules and tests — in the repository, diagrams and procedures — in the team's knowledge base. We support the first edit together with your engineers; after that you change the rules yourselves, without waiting for us. Describe your task — we'll get back to you with an estimate and a plan for such a package.

Case: assistant in the app before and after restrictions

What it was like before the restrictions

The assistant responded to free-form user requests without output validation: the model could give advice on medical dosage, promise a refund on behalf of the service, or reference a non-existent clause in the agreement.

This did not create a separate problem as long as the share of such responses remained within the margin of statistical error.

When the assistant gained access to order history and balance, every other unsafe response turned into a support ticket — the operator spent time reviewing the conversation and explaining that the bot had no right to promise that.

What was changed

We placed a layer of checks between the model and the app interface: input rules filter out requests outside permitted topics, output rules check the response for prohibited wording, promises made on behalf of the company, and references to data that is not in the user's context.

A response that fails validation is not shown — the assistant reformulates it or transfers the dialogue to an operator. All triggers are written to a log with the reason, so the rules are refined based on real cases rather than assumptions.

Metrics before and after

The load on support decreased not because users started writing less often, but because there were fewer reasons to write.

Separately, we note: the rules did not make the assistant useless — the share of dialogues resolved within the app grew, because the user stopped receiving responses that were later retracted.

We use Guardrails AI as a framework for describing rules and validators: it allows us to keep checks declarative and change them without rebuilding the assistant's logic.

How does input validation differ from output validation?

Input validation triggers before a request reaches the model: it filters out prohibited topics, attempts to extract the system instruction, suspicious attachments, and requests outside the application's scenario.

Output validation works with the already generated response: it catches fabricated facts, leaks of personal data, references to non-existent sections, and a tone unacceptable for the brand.

These are not duplicating but complementary control points — input is cheaper and faster, output is more precise because it sees the result in full.

False positives are distributed differently. At input they are usually coarser: a formally neutral question about medicine or finance may be filtered out as a prohibited topic, even though the user was asking about configuring notifications.

At output, false positives are subtler and more costly: the model produced a correct answer, but the filter mistook an order number that the user themselves provided in the request for a leak.

That is why thresholds at input are kept looser, and at output — stricter, but with mandatory logging of every trigger so an engineer can examine it and fine-tune the rule.

The validation loop is built so that control points do not duplicate each other. At input — request normalization, checking against prohibited categories, and limiting context length.

At output — verifying facts against a trusted source, masking personal data, and checking compliance with the scenario. If input filters out a request, the user receives a clear explanation and an alternative rather than a blank screen.

If output fails validation, the response is not shown but routed to a safe scenario: a clarifying question or handover to an operator.

The key decision is what to do when a trigger occurs. The soft option: flag the response and show it with a caveat. The hard option: block it and ask again.

The choice depends on the cost of an error in a specific scenario — in a reference section a caveat is sufficient, while in operations involving data, blocking is required.

Both levels are logged identically, otherwise it is impossible to calculate the share of false positives and understand where a rule is not working as intended.

Common technical concerns about guardrails ai

Before approving the architecture, technical teams ask the same questions: will response time increase, can the rules be bypassed, what happens when the model changes, and how to roll back an unsuccessful change. Below are short answers to each of these points, without marketing.

Can guardrails ai be bypassed by rephrasing the request?

It is impossible to eliminate this entirely — just like any protection. The goal is different: to make bypassing costly and noticeable. We combine rules on the input (intent classification) and on the output (response verification), plus we log bypass attempts.

When a threshold is triggered, the assistant responds with a refusal and hands the dialogue over to an operator. The rules are updated as new bypass scenarios emerge.

What happens when the model is updated?

Guardrails ai live separately from the model, so changing the version does not break the rules — but it does require running a regression.

We maintain a set of test dialogues on which we verify that the new model has not started bypassing the restrictions and has not lost quality on key scenarios. Only after passing the tests does the model go to production.

Can custom scenarios and rules be added?

Yes. The rules are described declaratively and do not require rewriting the application logic. You add a scenario — we configure the checks, test, and roll it out. Custom response formats, tone, and prohibited phrasings for your product are configured in the same way.

How do you roll back changes if a rule broke a scenario?

The rule configuration is versioned. A rollback is a return to the previous configuration version, not a rebuild of the application: it takes minutes and does not affect the release cycle. Before rollout, changes go through shadow traffic, so a rollback is rarely needed.

Describe your task — we will come back with an estimate and a plan for your product.

Let's discuss the task: we'll come back with an estimate and a plan

To come back with an estimate and a plan rather than generalities, we need baseline data about your application.

Bring what you have: the repository or access to it, the current architecture and data schema, the list of external services the assistant is integrated with, and a description of how it behaves now — where it answers off-topic, where it goes beyond the scenario, where users complain.

If the application is already in the stores, add analytics on the assistant's screens and an export of support requests: they show which answers break the scenario most often.

Next, we clarify the boundaries of the task. What exactly the assistant should do and what it is forbidden to do, what data it sees, where an answer must be escalated to a human, what checks are needed before showing an answer to the user.

Separately, we discuss how you will verify the result: a set of test scenarios, reference answers, metrics on the share of correct answers and on the number of out-of-scope outputs. Without this, an estimate turns into guesswork, and timelines into an endless process.

As a result, you get:

• an analysis of the assistant's current state and a list of places where it behaves unpredictably;
• a list of rules and constraints that need to be built into its operation;
• a scheme of checks for an answer before showing it to the user;
• a phased work plan with order and timelines;
• an estimate of effort for each phase;
• a list of risks and of what could shift the timelines;
• an option for rolling out changes without stopping the application.

We discuss the order of work in the very first conversation: first we analyze the task and the data, then we fix the scope, then we name the timelines. Describe the task in any convenient form — we'll come back with an estimate and a plan you can use to make a decision.