Why AI Data Analyst gets numbers wrong and breaks trust
The AI Data Analyst answers quickly and confidently, so its mistake looks like fact. A classic failure is incorrect aggregation: a metric is summed across rows that already contain a calculated total, and the number is doubled.
The user sees a plausible sum, cannot verify it against the accounting system, and stops trusting the tool entirely.
The second type is an empty result set. Errors in query generation in selection conditions (date type mismatch, an extra status filter, case sensitivity in string comparison) produce zero rows, while the interface shows a neat zero or an empty chart.
It is impossible to distinguish "there is no data" from "the data was not found" from the answer, and this is exactly where trust in numbers breaks most quietly.
Hidden duplicates appear on joins: if payment or contact rows are pulled into deals, each deal is multiplied, and revenue grows without a single change in the source.
Nearby is lost matching of reference data: "Romashka LLC" and "LLC Romashka" remain different clients, some categories drop out of the slice, and group totals stop reconciling with each other.
For the user, all these cases look the same — a beautiful answer with a chart that cannot be checked.
Therefore verification of answers is built not on trust in the wording, but on engineering rules: reconciliation with control totals, mandatory return of the number of rows involved, analysis of the generated query, and explicit signs of an empty or suspiciously short result set. Below we discuss how this is closed in the architecture of the AI Data Analyst itself.
Implementation effect: response times and reliability of numbers
Before implementation, the path from question to number went through a queue to an analyst: statement of the task, search for the needed data mart, manual query, verification, export to a table.
After launching the AI Data Analyst on Text-to-SQL, the same question is resolved in a dialogue — the system builds a query from the described data model and shows not only the result but also the logic of obtaining it.
What we record at acceptance. The time from question to a ready number for typical metrics drops from hours to seconds, and a repeated query no longer requires an engineer's involvement.
The share of questions closed without an analyst usually reaches 60–80%: the rest are heavy calculations, new sources, and disputed metric definitions that deliberately remain with the data team.
Manual exports for regular slices move to self-service, and the load on analysts shifts from "get the number" to "describe and refine the metric".
Reliability is ensured not by the model itself but by the verification loop around it. Each metric is described once: formula, source, filters, access rights.
A query is validated against the schema before execution, the answer is accompanied by a reference to the tables used, and all requests are logged.
Therefore the share of erroneous answers decreases, and discrepancies where the same indicator in two reports was calculated differently simply stop occurring.
Implementation options and their technical limitations
The choice of scheme is determined not by the model but by where the data lies and who can see it. The AI Data Analyst translates a question into the language of queries, and the deployment method determines where that query goes.
It is one thing to answer from an open set of metrics, and another to access a production data warehouse with personal data.
| Option | When it fits | What limits it |
|---|---|---|
| Ready-made analytics service | Need a result quickly, data is non-critical, team is small | Queries and data schema go to a third party; fine-tuning for your data model is unavailable |
| Deployment in a closed environment | Personal data, regulator requirements, prohibition on external calls | Requires your own capacity; model updates are on your side; launch takes longer |
| Hybrid scheme | Part of the data can be exposed externally, part cannot | Routing is more complex: you need to explicitly separate what goes outside the perimeter and monitor it |
| Semantic layer over the warehouse | You already have data marts and reporting; you need a question layer | Limited by the composition of data marts: what is not in the data model, the analyst will not see |
The choice comes down to three questions: what data enters the query, who updates the semantic model, and what to do when the answer diverges from the report. A combination is often used — the sensitive layer remains in the closed environment, and only the aggregate goes outside.
Work sequence: from source audit to handover
We conduct implementation in iterations, and each stage ends with an artifact that can be verified before moving to the next. The sequence below is the same for a pilot and for a production environment — only the volume of sources changes.
- Data audit. We inventory sources and collect a table profile: share of empty values, record duplicates, discrepancies in reference data, time zones in dates. The output is a list of fields suitable for analytics and a separate list of what requires cleaning on the side of the system owner.
- Solution design. We fix entities and relationships and build the semantic layer: metrics, dimensions, synonyms of business terms. We separately agree on disputed indicators — how to calculate revenue with returns, what an active client is, and what period is used for comparison.
- Access rights setup. We differentiate roles and row visibility, and enable logging of every query: who asked and what they received. A branch manager sees their own data and does not see others' — without manual exports and correspondence.
- Assembly and configuration. We connect sources, translate natural-language questions into database queries, and achieve resilience to different phrasings of the same question. Frequent requests are cached; heavy ones are calculated on a schedule.
- Acceptance on control questions. We prepare a set of several dozen real questions with reference answers and compare the result line by line. If there is a discrepancy, we look at whether the error is in the semantics or in the data, and fix it before signing.
- Handover and training. We provide documentation on the semantic layer, user instructions, analysis of typical errors, and a regulation: what we do when the source structure changes. We train the working group and support the first weeks of independent work.
What stays with you: configurations, schemas, regulations
The handover package is not a presentation or access to a demo stand, but a set of artifacts by which your team can support the AI Data Analyst without us. Each item is recorded in your repository and goes through acceptance together with your side.
- Data model description — connected sources, tables and relationships used, places where fields are interpreted ambiguously, and the decisions made on them.
- Metric dictionary — each business concept is tied to a formula, source, and filters; differences in interpretation between departments are visible at the setup stage, not in a finished report.
- Access policies — roles, row- and column-level differentiation, masking of personal data, request log: you can see who asked which question.
- Configuration handover — connection files, query processing parameters, dictionary update schedule, versioning of settings in your repository.
- Deployment scheme — where the service runs, how it is connected to the warehouse, what failure points exist, and how rollback to the previous version is performed.
- Set of verification questions — reference pairs "question → correct answer" used to check correctness after any model change.
- User instructions — how to formulate a question, what clarifications the system will request, and what to do if an answer looks doubtful.
- Update regulation — who edits the dictionary when the source schema changes, how changes are tested, and who confirms the release.
Case study: how we raised the share of correct answers from 62% to 94%
We review a project at a retail chain: the data lay in ClickHouse and several relational databases, and questions to them went through a queue of four analysts.
The answer to a typical question "how much was shipped in the region for the quarter" took up to three days — clarifying the statement, finding the data mart, writing the query, reconciling the number with the data owner. Out of a hundred requests per month, about twenty were returned for revision.
The starting point was a golden set: about 600 questions to the database with reference queries and labeling by data marts.
On it, the system gave 62% correct answers, and the rest was divided by incorrect table joins, lost period filters, and substitution of a metric by a similarly named one.
The time to prepare a number — from formulating the question to the confirmed figure — was measured as a separate metric.
What was changed
The main effect came from the semantic layer: descriptions of tables and columns, canonical keys, business terms tied to specific expressions.
A query does not go to the user until it passes a check for execution, allowed tables, and row limit; and if the database errors, the system parses the message and rebuilds the join.
Disputed cases — when there is no data in the sources or the metric is defined ambiguously — are marked as no answer, not as a guess.
What the result was
The share of correct answers grew from 62% to 94% on the same set; the remainder are out-of-domain questions, which the system honestly rejects. The average time to prepare a number decreased from two or three days to several minutes including source verification.
Manual exports fell from forty-plus to a few per month, and the class of data matching errors disappeared: client keys and returns are normalized before query construction, so double counting and incorrect joins by a non-canonical key are no longer reproduced.
How does an AI analyst differ from regular reporting?
Regular reporting answers a question formulated in advance: metrics are described, formulas are agreed, calculation runs on a schedule and gives the same number for the same period.
Its strength is repeatability: discrepancies between reports are visible immediately, and the result does not depend on who is looking: the report, filters, and order of slices are set in advance.
The AI Data Analyst works differently — it assembles a query to the data for a specific one-off question: how the segment's orders behaved after a tariff change, how many clients returned in two weeks.
No one designed such a report, but the answer is built not from nothing but from the semantic layer: table descriptions, relationships, metric dictionary, and filtering rules.
The difference is not in the data source — both approaches read the same data marts. The difference is in the moment when the question becomes precise: in reporting, at design time; for the analyst, at query time. Hence stability in one case and speed on non-standard slices in the other.
This is where the boundaries of applicability lie. A question that repeats every week belongs in regular reporting: it is cheaper to maintain. Running the AI Data Analyst for the same number means re-checking query correctness every time.
A one-off question on a narrow slice, conversely, is not worth putting into reporting: designing and maintaining the metric will not pay off.
The guideline is simple: the AI Data Analyst is reliable where data is described in the metric dictionary. Outside the described entities, it does not guess but reports what is missing — such a question goes to an engineer.
Concerns about data quality, access, and load
The analyst answers questions about data, so before implementation, what is checked is not the model but the state of the data itself and how the solution will fit into existing constraints: rights, load, schema changes. Below are the objections that most often come up in technical discussions and how we address them.
Reference data is dirty, some values were entered manually
Ideal data is not necessary. At the start, we fix normalization rules: we bring reference data to a single key, and we cut duplicates and service records at the level of views, not source tables.
If a value is ambiguous, the analyst reports this in the answer instead of substituting the first one that comes along.
Employees have different rights; not everyone can see the needed tables
The analyst works under the user's account, not a service one. Data to which a person does not have access will not physically enter the answer, even if they formulate the question differently.
We separate rights by roles and columns on the warehouse side — one configuration point instead of copies.
Will the load on the warehouse increase?
Queries go through a layer of views with limits on depth and result set size. Accesses to detailed tables are redirected to pre-calculated aggregates, so analytics does not compete with reporting for resources.
We control the load via the log: you can see which questions are the most expensive, and they can be optimized selectively.
The schema will change after implementation — everything will break
Schema changes are inevitable. The description layer lives separately from the warehouse: a column rename or a new table is edited there, not in code.
We keep a set of reference questions and answers — after each migration we run it and see, before users do, what has stopped answering correctly.
We address these questions at the design stage, before the first user queries — so they do not turn into production incidents.Let's discuss the task: what to prepare for the conversation
To make the estimate substantive rather than "plus or minus an order of magnitude," a few inputs are enough for us. They also become the basis for acceptance criteria: we then use them to check that the AI Data Analyst answers real employee questions, not demo ones.
- Task statement — what decisions are made based on the data and what is getting in the way now: slow exports, a queue to the analyst, reports assembled manually.
- List of sources — databases, warehouses, tables and views, metric dictionary; what is considered "truth" if numbers in reports diverge.
- Sample questions — 20–50 phrasings as employees actually ask them, with expected answers for at least some.
- Access rules — under which account the analyst works, which tables and columns are closed to it, and how personal data is masked.
- Boundaries of what is allowed — read-only, row limit in the answer, behavior for an ambiguous query: ask the user for clarification or refuse.
- Acceptance criteria — the share of questions from the set that are answered correctly, how we verify numeric matching, and what we consider an error.
- Operation — where we deploy, who the owner is, and how table descriptions are updated when the schema changes.
Send what you already have: a list of sources and a dozen live questions. We will come back with an estimate, a stage plan, and a set of checks for your data.