System prompt development: where the assistant answers incorrectly
Unstable assistant responses are almost never related to model capabilities — the cause lies in the instruction. The same question gets different answers under different phrasings, and the tone changes from one reply to the next.
System prompt development begins with acknowledging that we are dealing not with text but with a set of behavioral rules.
The most common mistake is contradictions inside the prompt for an AI assistant. One part requires short answers, another requires explaining the product in detail; one rule forbids mentioning competitors, another asks for a comparison. The model resolves the conflict anew in each dialogue — hence the instability.
The second source is loss of context in a long dialogue. By the tenth message, the assistant forgets that the client has already named the plan or region and asks the same questions again.
If the language model's behavior is not described for repeated clarifications and topic changes, the conversation falls apart into unrelated answers.
The third is the absence of priorities. Without an explicit order of "facts first, style second," any nonstandard request breaks the scenario. In support, this means invented deadlines and terms; in sales, a missed clarifying question and a lost contact.
Such errors in the assistant instruction are reproduced on a set of test dialogues rather than guessed. Next, we examine how we assemble and test the system prompt.
Measurable effect of a refined assistant instruction
The effect of a refined instruction is measured on the same reference set of dialogues — several hundred scenarios sorted by type: a standard request, a clarification during the conversation, a request outside the assistant's area of competence, and contradictory requirements in a single message.
Evaluation is blind: annotators do not know which version of the instruction responded, and disagreements between them are counted separately — otherwise annotation noise can easily be mistaken for improvement.
The first metric is the share of correct answers.
After cleaning up the rules and examples, it grows from 70–75 to 90+ percent on the same scenarios, while the number of escalations to an operator on them drops severalfold: the assistant stops retreating into general wording and drives the dialogue to a result.
We look separately at answers outside its competence — there the metric is not "right/wrong" but the correctness of escalation.
Repeatability is checked by running the set several times with fixed model generation parameters on the OpenAI side.
Before refinement, the spread in the share of correct answers between runs reached 10–15 pp; after, it stays within single digits: the result is reproducible rather than dependent on a lucky run.
A unified answer style is also a measurable quantity: structure, length, form of address, and refusal wording are set by explicit rules and checked automatically by markers, so a new scenario does not "drift apart" in tone from the rest.
The main change is predictability of refusals. The class of "confident incorrect answer" disappears; instead comes a clear refusal or handoff to an operator with ready context. Next, we examine how this instruction is assembled and kept in working order.
Ways to set assistant behavior: comparison of approaches
Assistant behavior is set by text inserted into every request before the user's message. The structure of this text determines how predictably the assistant maintains its role, answer format, and boundaries of permissible actions.
Below are four working variants of assistant instructions that teams actually choose between, and the limitations of each.
System prompt development is not a search for the "right" format but fitting the structure to the scenario. A short instruction often becomes the core, over which an assistant behavior policy is layered for separate branches.
We test the assembled scheme on a set of reference dialogues: we count the share of correct answers and refusals before and after edits.
| Variant | When it fits | Limitations |
|---|---|---|
| Short instruction: role, tone, answer format | One type of request, simple scenario | Loses exceptions — the assistant improvises disputed cases |
| Instruction with examples of exemplary answers | A stable format is needed: fields, markup, template | Request size grows, examples push toward copying |
| Behavior policy: rules, prohibitions, branches | Many constraints and different types of requests | Without a strict structure, some rules fall out of the model's attention |
| Multi-level scheme: core plus scenarios | Several scenarios with shared and their own rules | Harder debugging — level conflicts take longer to find |
The choice of variant is tested on dialogues that already broke the previous version: if the scheme does not maintain the required share of correct answers, it is simplified rather than supplemented with new rules.
Workflow for developing an assistant instruction
An assistant instruction is not a paragraph of politeness but a set of rules with priorities: what to do when instructions conflict, where to stay silent and hand the dialogue to a human, and which answer format is mandatory.
Therefore, the stages of instruction development begin not with text but with analyzing how the assistant currently answers. Below is the route from audit to handoff of a configured setup with tested scenarios.
- Audit of assistant answers: we annotate real dialogues — where it goes beyond its role, invents facts, breaks format. The output is a list of typical failures with frequency.
- Rule scheme: role, tone, competence boundaries, priorities when instructions conflict, and behavior when data is insufficient — to whom to hand off the request.
- Draft instruction: wording for each rule plus separate blocks for frequent scenarios so the main text does not grow too large.
- Test set: reference messages and expected answers, including provocations, fragmentary questions, and out-of-scope topics.
- Instruction testing: run through the set and comparison with the reference, revision of wording, version locking. Disputed answers are analyzed separately.
- Acceptance on the customer's scenarios: we clarify edge cases and answer tone.
- Handoff: instruction, test set, and a short update policy — when the product changes, rules are edited surgically.
After that, the instruction lives as a configuration: changes pass through the same test set, and regressions are visible before release rather than in a dialogue with a client.
What artifacts the team has after the project
The project ends not with a dialogue demo but with a package of files by which the assistant can be developed without us.
Without this, the instruction text lives in chat: edits are made "from memory," and after a change of owner no one can explain why a rule was worded exactly that way and what will break if it is removed.
- System instruction text — the final version with role separation, topic boundaries, and answer format, in a repository rather than in chat.
- Map of rules and priorities — which instructions take precedence in a conflict, when the assistant clarifies, and when it refuses.
- Set of verification scenarios — standard and edge-case dialogues on which every instruction edit is run.
- Testing rules — run order, reference answers, and the allowable share of discrepancies before a version is released.
- Update policy — owner of the instruction, procedure for making edits, and mandatory recording of the reason for each change.
- Version history — what was changed, why, on which set of scenarios it was tested, and with what result.
- Assistant documentation — roles, limitations, connected knowledge sources, and known weaknesses.
- Handoff checklist — accesses, environments, procedure for launching checks, and escalation points for deviations.
This set covers the main risk: the logic of the assistant's behavior remains with the customer, not the contractor. Updating the instruction becomes a clear procedure with testing rather than an experiment on live users.
Case study: how discrepancies in assistant answers were eliminated
The assistant of the ServicePro service center network answered customers in chat, email, and messenger.
On a control set of 200 scenarios, the same question about deadlines and returns produced discrepancies in 29% of cases: the rules were duplicated in three places in the instruction and contradicted each other.
This is not cosmetic — every third incorrect wording led to a repeat contact, and operators corrected answers manually. Below is exactly what was rewritten and how it affected the metrics.
What was wrong
Priority between rules was not set, so the model chose wording arbitrarily. The instruction said nothing about missing data, and the assistant filled in deadlines by analogy with similar requests; tone also fluctuated across channels.
What was rewritten and what was achieved
The instruction was broken into layers with explicit priority: facts from the knowledge base, then domain rules, then format and tone — in a conflict, the narrower rule wins.
A hard constraint was introduced: a deadline or condition without source support is not inferred; instead, the assistant asks a clarifying question or hands off to an operator. Disputed points were covered with "right / wrong" example pairs.
The same set after edits: discrepancies 29% → 2%, answers with invented detaпо индивидуальному расчёту% → 0.8%, share of correct answers 71% → 96%. The set became a regression suite — every instruction edit is run in full, and rollback to the previous version takes minutes.
What to do if the prompt is already written but answers fluctuate
Instability of answers is almost never related to a weak model — the source is the instruction itself.
If one block requires answering briefly while another insistently asks to reveal details, the assistant chooses differently each time: the outcome depends on which fragment of the rules ended up closer to the request. Until such a conflict is found, any wording edit produces an effect for a day.
We start diagnostics not by rewriting the prompt but by analyzing the answers that "drifted." We collect real requests where behavior diverges and sort them into three classes: the rule is not described at all; the rule exists but without priority; the rule is overridden by an example or output format.
Most often the second surfaces — several equivalent instructions with no order of precedence between them.
If one or two scenarios diverge, it is a prompt refinement: remove duplicate requirements, set explicit priority, move answer format into a separate block.
But when a noticeable share of requests "fluctuates" and across different topics at once, point edits only confuse — rules are described in a jumble, role and tone are set in different places and argue with each other. Here the instruction needs to be reassembled entirely.
Reassembly means breaking the role, permissible boundaries, priority order, and behavior rules for insufficient data into isolated blocks and fixing what matters more in a collision. Then the instruction is run on the same set of requests: before and after.
If the spread of answers on this set stops changing from run to run, the instruction is ready for work, not for the next series of edits.
Common technical doubts before starting
Doubts before starting are usually not about whether the model "can" but about controllability: will the instruction hold up as the number of scenarios grows, what happens after an update, who is responsible for an error in an answer. Below are the questions asked most often and what engineering lies behind them.
We treat the assistant's system instruction as a configuration rather than text: it has a version, a set of checks, and a change policy. Therefore, questions about maintenance have a procedural answer rather than a promise.
Will the instruction become unmanageable if many rules are added?
Rule density is a real limitation: after a certain threshold, the model begins to blur priorities between them. Therefore, we break rules into layers — role, facts, prohibitions, answer format — and test conflicting pairs on a set of dialogues.
If a new rule contradicts an existing one, the test shows this before release, not in a conversation with a client.
How can we make sure answers comply with the rules?
To test the assistant, we assemble a set of messages: standard, edge-case, and provocative. We run every version of the instruction through it and compare with the behavior reference — where the model departed from the policy, where it added extra.
We separately look at an incomplete question and an attempt to steer the conversation beyond the topic.
Is updating the instruction a full reassembly of the solution?
No. The layers are separated, so editing rules or tone does not affect connected data sources or integration logic. We change the needed layer, run checks, release — the previous version is preserved, and rollback takes minutes.
What does the customer receive at project handoff?
The instruction with version history, a set of verification dialogues, a description of layers, and a change policy: who edits the rules and how, and what is run before release.
Handoff does not require our participation — the team continues work according to the documented procedure.
Let's discuss your task: what to prepare for the conversation
To make the conversation substantive, a ready document with requirements is not needed — fragments are enough: chats, examples of unsuccessful answers, draft instructions.
The more concrete the source materials, the more accurate the project estimate and the fewer assumptions in the work plan.
- 5–10 real dialogues: both successful ones and those where the assistant went off track.
- Scenarios where the answer must be strictly defined and where freedom of wording is acceptable.
- Glossary and tone rules: what we call products, what we tell the client, and what we do not say.
- Hard constraints: topics on which the assistant stays silent and hands the dialogue to a human.
- 3–5 reference answers — we use them to configure acceptance criteria.
- Knowledge base: policies, tables, documents from which the assistant takes facts.
- Launch channels and roles: where the assistant works, who uses it, and who views dialogue history.
- Metrics: share of answers without operator edits, share of escalations, first response time.
In response, we come back with a project estimate and work plan: scope, stages, and what will be required from your side at each. We separately discuss acceptance criteria — you will use them to accept the result, not general impressions.
Preparation for kickoff takes little: access to the knowledge base, a test environment, and one person responsible for acceptance on your side. This is enough to start and not drag out approvals.