Implementing Chain-of-Thought (CoT) Prompting for LLMs

Chain-of-Thought Prompting: Why Answers Are Unstable

On multi-step tasks — reviewing a contract clause by clause, calculating under dependent conditions, diagnosing from logs — the model produces a coherent result that contradicts part of the source data.

Editing the wording helps for one or two runs, then the behavior diverges again: Chain-of-Thought prompting without chain-of-thought control does not provide stability.

The reason lies in the generation method itself: the answer is assembled step by step, and attention at each step is distributed across the entire request.

The longer the input, the higher the chance that a specific condition — a deadline constraint, an exception to a rule, a unit of measurement — will not make it into the current step. The chain of reasoning is not built: the model takes a short path to a plausible conclusion.

This is why model answers are unstable: the same request across different runs produces different reasoning branches and different conclusions.

Testing on one successful example says nothing about answer quality on complex tasks — divergence appears over dozens of runs and where the cost of an error is highest.

Loss of conditions in a request is not a wording defect but a property of generation itself, so rewording the question does not fix it. A controllable chain is needed: explicit steps, verification of intermediate results, and rules by which the model returns to the source data. We explain how this is implemented.

What Controllable Reasoning Delivers: Measurable Effects

Controllable reasoning is not a “smarter model” but a pre-described order of steps that the model follows before the conclusion. On multi-step tasks, this changes answer accuracy, repeatability, and the volume of manual review.

Share of Correct Answers on Multi-Step Tasks

Where an answer is assembled from several sequential inferences — checking conditions, processing a request according to regulations, calculation by several rules — the model without a defined structure loses intermediate conditions and produces a plausible but incorrect result.

When the template includes the sequence “extract facts → apply rule → verify result,” the share of correct answers on such sets increases severalfold.

Repeatability: Same Input — Predictable Output

Without a fixed schema, the same request yields different answers: arguments, order, sometimes the conclusion change — and the result can neither be embedded into automation nor accepted by a formal criterion.

We define mandatory steps, their order, and the criterion by which a conclusion is considered justified — the divergence between runs is reduced to wording, not meaning.

How Much Manual Review Is Removed

Manual review usually covers the entire output because individual answers cannot be trusted.

After setting up templates and a set of validation examples, an operator looks only at low-confidence branches and disputed cases — the volume of proofreading decreases, and along with it goes the class of errors “confident answer without data support,” which previously only a human could catch.

Reasoning Modes: Comparison and Limitations

Reasoning mode is a budget for intermediate steps, not the model’s “intelligence level.” The more steps between question and answer, the fewer errors on multi-condition tasks — and the higher the latency. We select the mode for the task, not “to the maximum.”

Below are four modes and their limitations.

When Direct Answer Is Enough

Direct answer works where the result is a single value or a short label: extract details from a contract, label a request by topic, bring text to a template. Extra steps add nothing — the model already hits the target, while response time grows. The criterion is simple: if a person answers without thinking, no reasoning is needed.

When Verification by a Second Pass Is Needed

A second pass is needed when an error is more expensive than latency: calculations, choosing among options, document-based answers. We separate the model’s reasoning and self-check into different calls: first the conclusion, then verification against source data and rules.

This catches an error in the middle of the chain and reduces the share of incorrect answers, but noticeably increases cost.

Workflow: From Task Analysis to Handover

Implementation begins not with model selection but with task analysis: where an answer must be accompanied by step-by-step reasoning, and where a short form is enough and Chain-of-Thought prompting will only slow the response.

We arrange the stages so that each next step relies on a verifiable artifact, not on the impression from a showcase example.

1. Task analysis and collection of real requests: we export the history of inquiries, remove duplicates and group by types — we get a set reflecting the actual distribution of tasks, not cases convenient for demonstration.
2. Labeling of reference reasoning: for each task type, we describe the chain of steps that leads to the correct answer.
3. Assembly of draft prompt sets: we formulate several variants for each task type and compare them on the same examples.
4. Setup of verification rules: we record what we consider a correct answer and how we catch reasoning that leads to an incorrect conclusion.
5. Run on a holdout set: only examples that did not participate in setup show the real stability of the solution.
6. Handover of the solution and team training: documentation, analysis of typical errors, and a regulation for updating prompt sets.

Labeling Reference Reasoning

What matters here is not the length of the chain but its reproducibility: two labelers on one example must arrive at the same answer. Therefore we put disputed cases into a separate list and resolve them before they enter the set.

Verification on a Holdout Set

The holdout set is collected in advance and is not used during setup — otherwise the evaluation shows memorization, not quality. We look at the share of correct answers for each task type separately: the overall percentage hides failures on rare but important requests.

What Remains with You: Prompt Library and Regulations

After the solution is launched, you retain not only a working service but also the entire layer that manages it.

The prompt library, verification regulation, and solution documentation are handed over to your repository — in the same form in which we use them ourselves, without “magic” in someone else’s infrastructure.

• Prompt sets by task type: purpose, expected answer format, examples of correct and rejected model reactions, including chains of reasoning.
• Schemes for analyzing typical tasks: which reasoning steps the model goes through, where the branching point is, and what is considered an incorrect conclusion.
• Verification regulation: rules for checking an answer against the source, a set of control requests, acceptance criteria for a new version.
• Update rules: what is reassembled when the model or provider changes, and what is transferred without edits.
• Change log: date, author, reason for the edit, result of the control set run before and after.
• Maintenance instructions: how to add a new scenario without breaking existing ones, and where to look when quality drops.

How Prompt Sets Are Stored and Updated

Prompt sets are kept in a version control system next to the service code: one prompt — one file with a description of its purpose. An edit enters the main branch only after a run of the control set, and the model version is recorded in the change log. When it is updated, you can see which scenarios dropped and what exactly was fixed.

Case: How Answer Accuracy Grew in Request Analysis

What It Was Before

Analysis of incoming inquiries was done with a single request: the model immediately produced a category and a decision.

On ambiguous formulations — and this is a noticeable part of a stream of several thousand inquiries per month — the conclusion turned out incorrect, and such errors were not caught by rules: there were not enough intermediate steps where the model compares facts with each other.

The share of correct conclusions stayed around two-thirds, and every third inquiry went to manual verification — up to 12 minutes of a specialist’s work.

Rules were added on top, but this treated symptoms: each new customer formulation required another exception, the list grew, and accuracy stayed in place.

What It Became After

Analysis was broken into steps: first extraction of facts from the inquiry text, then search for contradictions between them, then verification against the rules base — and only after that the final conclusion.

The reference corpus of 400 labeled inquiries was assembled in Label Studio, and every prompt version was run on it before rollout.

• share of correct conclusions — 94% versus 68% before rework;
• manual correction — one inquiry out of fifteen instead of every third;
• analysis time decreased from 12 to 4 minutes per inquiry;
• escalations to a senior specialist — three times less often.

The prompt lives as code: a change goes through a run on the reference set, and if the metric drops, the version is rolled back. New types of formulations are handled without editing rules — the model completes the chain of reasoning on its own.

How Does Step-by-Step Reasoning Differ from Self-Check?

Step-by-step reasoning is one pass: the model breaks the task into intermediate steps and follows them to the answer.

On arithmetic, analysis of conditions, and fact extraction, the share of correct answers grows noticeably, but the output lengthens — the analysis takes up the main part, while the result itself remains at the end.

Self-check is structured differently: the draft solution and its verification are different passes. The model first answers, then separately rereads the answer against the task criteria and corrects the places it finds.

Separating passes is needed where an error in the middle of the chain is invisible from the outside: one incorrect step produces a confidently looking but incorrect result, and without a second pass it cannot be caught.

Each additional pass increases processing time and the volume of calls to the model, and if the verification is shown in full, also the length of the answer. On short tasks, two passes instead of one noticeably add latency for the same result.

On long multi-step scenarios, the difference pays off: the share of answers that are later corrected manually falls.

We choose the mode by the cost of an error, not by the external complexity of the task. Where an incorrect answer is visible immediately and is easy for a human to check, step-by-step reasoning is enough.

Where the solution goes into an automatic loop — calculations, coordination of conditions, conclusions from documents — we add self-check, sometimes only on critical steps, so as not to pay for checking the entire answer.

Self-check can also be wrong, so we describe verification criteria and behavior in case of discrepancy in the prompt in advance.

Common Technical Doubts and Limitations

CoT is not a universal technique: on classification, field extraction, and compliance checking, step-by-step reasoning increases response time and output volume without quality gains.

We mark in advance where the method gives an effect and where it should not be enabled — this is part of project work, not “by eye” tuning.

Below are technical doubts that most often come up at the evaluation stage: stability when changing the model, the need for reasoning examples, measurability of the result, and transfer of prompts to neighboring scenarios.

Are reasoning examples in a prompt mandatory?

No, step-by-step reasoning works without labeled examples too. Examples speed up reaching the required quality and stabilize the format, but lengthen the prompt and increase output volume by 2-4 times.

Therefore we deliver the result in a specified format, while leaving the reasoning process internal — the user does not see extra text.

How to measure the effect instead of taking it on faith?

We assemble a reference set of requests with pre-known correct answers and calculate the share of correct solutions before and after. Additionally, we look at the share of refusals and human escalations. Without a reference set, comparison remains a matter of taste.

Can prompts be transferred to another scenario?

Partially: the reasoning structure transfers, specific formulations do not. We keep prompts as a separate layer with versions and tests, so transferring to a new task is adaptation and a repeated run, not development from scratch.

Let’s Discuss the Task: What to Clarify Before Start

An accurate estimate for CoT prompting begins not with a prompt template but with input data. If you send only a task description, we will come back with a range several times wider than the real one.

Below is the minimum set for an estimate, after which the conversation moves to a work plan with stages and timelines.

• Examples of requests — 30-50 real formulations, including short ones, with typos and mixed languages.
• Reference answers — for 20-30 examples, the recorded desired result and what is considered unacceptable.
• Typical failures — where the current logic makes mistakes: a missed reasoning step, substitution of a fact, truncation on a long request.
• Verification requirements — which checks an answer passes before output: comparison with the source, format, mandatory fields.
• Response time constraints — how many seconds the user waits and what is more important: speed or depth of reasoning.
• Data constraints — what cannot be transferred to external services, whether environment isolation is needed.
• Integration points — where the request comes from, where the answer goes, who edits the template after launch.
• Target metrics — share of correct answers on the validation set, share of “I don’t know” refusals, variance between runs.

This set starts the task discussion: we hold a project call with an engineer, analyze disputed examples, record what we consider the reference, and remove contradictions in requirements.

Based on the results, we send a work plan — the order of stages, the composition of checks, documentation, and a timeline estimate.