Development of an AI conversational search system with contextual understanding

Why conversational search breaks on real queries

In conversational search, the meaning of a query almost never lives in a single turn.

A user asks "what if it's for two branches?" — and the entire substance of the question is left behind in the previous two messages: which product, which plan, what has already been clarified about timelines.

A system that processes each turn as a standalone query sees only a set of words with no anchor and returns the closest match by words, not by meaning.

The main breakdown in dialogue context occurs on follow-up queries and pronouns. "How much does this cost", "is there anything cheaper", "will it work for reporting" — there is not a single term here to latch onto directly.

Add a topic switch in the middle of the conversation and a return to an earlier question — and similarity matching starts blending topics, pulling in fragments from unrelated turns.

The cost of the error shows up in the metrics: the share of irrelevant answers grows noticeably faster on long dialogues than on short ones, because every subsequent step accumulates distortion.

The user rephrases three or four times, drops into support, or abandons the scenario — retrieval quality falls exactly where the dialogue was supposed to save their time.

A separate cause is the absence of an explicit conversation state model: the system does not store what has already been established and does not distinguish when a clarification changes the query and when it merely supplements it.

So let us break down how this is closed: which layer is responsible for rewriting the query with history taken into account, where answer-relevance checking comes in, and how to measure that the dialogue has actually become shorter.

The effect of contextual understanding: metrics and failure classes

We compute metrics not by feel but on a reference set of dialogues: several hundred real conversations, annotated step by step — what was asked, what was meant, and which answer counts as correct.

The run is performed on shadow traffic, so the share of unsuccessful answers is measured on the same queries that reach the product. We keep annotation and scoring on the open evaluation metrics set RAGAS — the scale is the same before and after, so the comparison is valid.

The main shift is visible in the structure of failures, not in the average.

The class where the system answers the last turn in isolation from the preceding ones (a lost pronoun referent, entity substitution, an answer from an already closed branch of the conversation) shrinks severalfold with contextual understanding — it was precisely this class that produced the most unsuccessful answers in multi-step dialogues.

Failures of the "did not understand the wording" type remain, but they stop masking context loss and are visible separately.

Response time does not grow linearly: context assembly is a separate step before retrieval. To stay within budget, we limit reranking depth, cache "turn + context" pairs, and prevent the reformulation from expanding into multiple passes.

At the output, what matters more is the tail rather than the average: the gap between the median and the worst answers narrows, because heavy multi-step dialogues no longer take the slowest path.

The load on operators is a derived metric. When the system retains context, the user does not have to re-enter the background, and the share of escalations and repeat contacts on the same topic falls.

We also track the inverse: how many clarifying questions the system asks before answering — an unnecessary clarification drives the user out of the dialogue just as a wrong answer does.

We verify by comparison on a single sample: the version without context and the version with it run in parallel, and we take metrics daily. That shows where the shift is stable and where it is sample noise.

We analyse the remaining failure classes by cause rather than by the fact that "the answer is bad" — and that is already work on the next quality increment.

Conversational search modes and when they are appropriate

The retrieval mode determines how the system matches the reformulated query against the knowledge base. The conversation adds context: for the turn "what if there's no prepayment?", lexical search without substituting the previous topic will return random documents.

That is why the query is first rewritten taking the dialogue history and pronoun references into account, and only then enters a specific retrieval mode.

The choice of mode is not a matter of taste but a consequence of the query distribution and the cost of error.

The limitations show up at the edges: if a noticeable share of turns in the logs contains negations or exact codes, pure semantics will cut them off, and if the dialogues are descriptive, lexical search will find nothing.

We look at an annotated sample of real dialogues, compute the top-3 hit rate, and only then lock in the scheme; in most projects the starting point is a hybrid mode, and we enable reranking selectively where the cost of error exceeds the cost of latency.

Implementation stages: dialogue audit, scheme, tuning, handover

You cannot tune conversational search using demo queries: real turns are shorter, come with typos and pronouns — "is there a cheaper one?", "the same but with delivery". Until the distribution of such queries is visible, any scheme remains a hypothesis. That is why we start with a dialogue audit and finish by handing the policies over to your team.

1. Dialogue audit. We export conversation logs and annotate where retrieval loses context: anaphora, omitted subject, topic switch within a session. The output is the share of turns that lack the preceding one.
2. Context scheme. We determine what lives in the session: recent turns, extracted entities, selected filters. Separately, the reset condition — otherwise the clarification "no, not that one" drags the wrong result along with it.
3. Retrieval tuning. Ranking, relevance thresholds, priority of the fresh turn over history, rules for merging filters and resolving conflicts between them.
4. Validation on live queries. A run on a held-out sample of real dialogues rather than on a test set: the top-3 hit rate and how often a second attempt is enough for the user.
5. Pilot on a share of traffic. We compare against the current search on the same audience and review the dialogues where it got worse — usually a change of intent in the middle of a session.
6. Handover of policies. The context scheme, instructions for annotating new dialogues, a quality dashboard, and the procedure for extending the rules with your own team.

After the handover, your specialists change thresholds and retrieval rules without rewriting the core. That is the readiness criterion: the system evolves within the team, not through us.

What we hand over: configurations, schemes, policies

The system must not remain a "black box" that only its author can maintain. We hand over a set of artefacts that lets your team change answer scenarios, connect new sources, and deploy the environment from scratch. Below is exactly what remains with you after acceptance.

• System configurations — search and ranking setting files: field weights, relevance thresholds, the depth of dialogue context taken into account, and rules for splitting documents into chunks. They are edited without code changes, with a clear description of each parameter and its effect on the output.
• Data flow schemes — the path of content from source to index: ingestion, normalisation, cleaning, annotation; plus a separate scheme for query flow — from the user's wording through context refinement to a finished answer with a link to the primary source.
• Access policies — role and collection separation: which sources each user group sees, how permissions are checked at the moment the answer is served, and how quoting from restricted documents is limited.
• Documentation — a description of the architecture and the purpose of each node, the exchange formats between them, local run instructions, and typical faults with their symptoms.
• Update policies — the procedure for rolling out a new model branch, rebuilding the index, and editing scenarios: which checks run before release and at what quality deviations the release is rolled back.
• A set of validation questions with reference answers — used after any change to see whether quality has dropped on key topics, and in which scenarios exactly.
• Monitoring policy — which metrics to watch, what counts as quality degradation, who to notify about a failure, and in what order to act.

The handover is not an archive drop but a walkthrough: we show what is where, make the first changes together with your engineers, and answer questions along the way.

After that, evolving conversational search — adding sources, retraining the ranking, new channels — does not require coming back to us.

Case study: how we cut the share of unsuccessful answers threefold

The client is a support service with a distributed first line and a knowledge base of several thousand articles on Zendesk.

Operators searched for answers by keywords: the customer's wording almost never matched the article's wording, and the context of preceding turns in the dialogue was not taken into account at all.

It broke down in three places. Search returned a neighbouring article on the topic, the operator went into manual investigation or answered at random — and the customer came back with the same question.

In long dialogues the connection to previous messages was lost, so clarifications had to be collected from scratch.

What we changed

We built conversational search on top of the knowledge base: the query is parsed with the dialogue history taken into account, and the answer is formed only from retrieved fragments and with a link to the source.

When confidence is low, the system does not guess but hands the dialogue to an operator together with the assembled context — escalation became deliberate rather than emergency-driven.

Metrics before and after

The comparison is made on a single set of questions annotated manually before and after implementation — the growth in the share of correct answers is not attributed to seasonality.

In addition, we monitored a shift for a month: abandonments decreased and repeat contacts became isolated. The result of the project is search that the operator trusts and relies on directly during the conversation.

How contextual understanding differs from history substitution

Dialogue history substitution is the mechanical concatenation of recent turns into a single query.

The technique works over two or three steps, but in a multi-step dialogue it produces context loss: a key early turn is displaced by a fresh one, and the model answers the last phrase rather than the user's intent.

The window grows while accuracy falls — because the volume of history is not equal to the structure of meaning.

The reason lies in the nature of live speech. Turns are full of ellipses and pointers without an antecedent: "what if it's the other way round", "compare with the previous one", "the same filter but broader".

If the state of the conversation is not explicitly recorded anywhere, there is nothing to bind these pointers to, and the search query remains incomplete.

Contextual understanding differs in that each new step is reformulated into a standalone query with restored entities and constraints.

Engineering-wise, this is moved into a separate state layer: the active topic and subtopic, refined conditions, dismissed and rejected options, and the return point when the branch changes.

The layer updates on every turn and passes a ready formulation to search, not a retelling of the chat log.

The retrieval is hybrid — a vector based on semantic representations of the text plus lexical keyword matching in Qdrant, so that rare terms and SKUs are not lost behind generalisation.

Topic retention is tested with provocative dialogues: switching subtopic and returning to the original one, adding a clarification, negating something said earlier. This set is run after every logic change, and the share of steps where meaning is preserved becomes visible.

It is that share, not the prompt length, that determines whether the system can sustain a ten-step conversation.

Technical concerns: terminology, data, integrations

The client's technical specialists ask the same questions at acceptance — and they are fair: the system works with their documents and has to fit into their environment. Let us address the concerns that come up most often.

How are unsupported inferences restricted and sources controlled?

The answer is built only on indexed documents, and every statement is accompanied by a link to the primary source with a section number. If there is no confirmation, the system says so directly rather than filling in the answer with guesses.

The confidence threshold is configurable: you can allow an answer only when it matches two independent sources.

Will an employee see documents they do not have access to?

The exchange runs through a service account, and permissions are inherited from your system: only what a specific employee is allowed to see appears in the output. The same rule applies when exporting question history for review.

Can we fine-tune the system ourselves?

Yes. Dictionaries, citation rules, and answer-format prompts are moved into a separate layer that can be edited without us. Your engineer adds a term or closes off an entire topic — the change takes effect the same day, without rebuilding the model or repeating acceptance.

Let us discuss your task and come back with a work plan

To keep the discussion concrete, we start with your scenario rather than general questions: how the user formulates a query, what clarifications they make in the second or third message, and what they consider a good answer.

Real examples immediately show where conversational search loses context — on pronoun substitution, on an entity from the previous turn, or on a topic switch.

What we need from you for an estimate:

• 20-30 real dialogues with the expected answer — we use them to check how the system retains history and clarifications.
• Where the data lives and how often it changes: an export, access to the search index, or queries to your system.
• What must not be shown or stored: anonymisation rules, dialogue history retention period.
• Deployment requirements: your environment, an isolated network segment, or an external platform.

What we will return:

• A stage-by-stage work plan with acceptance checkpoints and quality metrics for each.
• A timeline estimate: when the first working environment launches and when we reach the target accuracy.
• Team composition: who owns the dialogue logic, who owns the search index, who owns deployment.
• The next step is a short meeting on your examples, after which we lock in the stages and start.

Describe the task in two or three paragraphs and attach example queries — we will come back with an estimate and a work plan.