Why BI AI Copilot does not answer business questions
An implemented assistant more often answers not incorrectly, but uncertainly: the number arrives, but it does not reconcile with the report, and the wording changes from question to question.
The reason is almost never in the model itself — the issue is that beneath it there is no layer where metric definitions are fixed unambiguously.
The metric “revenue” lives in three or four places: a data mart, a report in Power BI, a table in analytics, an export for the planning department. Somewhere it is the amount including tax, somewhere excluding it, somewhere it is shipment, somewhere payment.
BI AI Copilot takes the nearest available source and answers correctly relative to it — but diverges from the number a person sees in their report.
Then language comes into play. “Margin,” “active customers,” “revenue by region” — slang and abbreviations are not mapped to the data model, so the assistant guesses the field by string matching.
Contradictory metric definitions are not visible in the answer or the interface: the user gets a number without a caveat that the question was understood in its own way.
Separately — the query context: filters, granularity, time zone, permission level. It is not passed further, and answers to metric questions are calculated across the entire database — either diverging from the report or showing what the user was not supposed to see.
Trust in reporting falls faster than coverage grows: after two or three discrepancies, the analyst returns to familiar exports. We break down how this is closed — with a single layer of definitions and rules by which the assistant answers.
What AI analytics changes in working with metrics
The first shift is time to answer. Previously, the question “why did conversion drop in a specific channel last week” went into the analyst’s queue: request, clarification, export, consolidation.
Now BI AI Copilot answers in minutes because it builds a query to the semantic layer over the warehouse rather than rebuilding the report from scratch.
The second is the share of manual exports. Summaries that were assembled by copying from several systems collapse into a query to unified metric definitions.
One metric is calculated once: the same in the reporting dashboard, in the assistant’s answer, and in the regular distribution.
The third is the load on the analytics team. Recurring requests (“give me a breakdown by region,” “compare month over month”) stop reaching people. Engineers work on the data model, source quality, and new data marts instead of endless exports for the same question.
The fourth, the most expensive, is the eliminated class of reporting errors. Manual assembly produced double counting, an outdated period, and diverging definitions of the same metric.
The assistant returns not a number but the path to it: query, semantic version, period, source — the answer is reproducible and verifiable.
In sum, what changes is not the speed of an individual report, but the mode of working with data: a question is asked when it arises, not when its turn comes.
Connection methods: direct access, semantic layer, data marts
The connection method determines what an AI assistant (BI AI Copilot) will even see and how reproducible its answers are.
There are three options, and in practice they do not compete but complement each other: data marts cover recurring questions, the semantic layer covers free-form wording, direct access covers one-off investigations across heterogeneous sources.
| Option | When it fits | Limitations |
|---|---|---|
| Direct access to the warehouse | One-off investigations, heterogeneous sources, an analyst nearby to verify answers | The assistant infers relationships between tables itself, some aggregates are calculated incorrectly; broad scans over raw data; access rights are limited at the schema level |
| Semantic layer | Metrics are already defined, and the same answer is needed from different reports | It depends on the metric catalog: without formulas, granularity, and an owner, the layer remains formal |
| Materialized data marts | Recurring questions on a stable set of metrics | Freshness is limited by the rebuild schedule; a new dimension requires modifying the data mart |
The difference between the options is not in answer speed, but in the requirements for metric descriptions and in who is responsible for discrepancies in numbers. The defining element is the metric catalog: calculation formula, granularity, owner, permissible dimensions.
Until it is filled in, any connection method gives a confident answer without a basis, and that is more dangerous than no assistant at all.
Implementation stages: from metric audit to handover to the team
Implementing BI AI Copilot is not installing a tool but a sequence of engineering work where each step relies on the result of the previous one. Until it is described where each number comes from, any assistant answer remains an assumption. Below is the order in which we run the project.
- Data source audit: we compile a list of warehouses, data marts, and exports that actually feed reporting, record the owner of each source and discrepancies between systems — for example, when revenue in billing and in the sales report differs by the amount of returns.
- Business metric description: for each one we define the calculation formula, granularity, period, and exclusion rules — test orders, internal accounts, reversals. A metric is considered described when its value matches the reference report for control periods.
- Access rights setup: we inherit roles from the data source rather than creating a parallel model. The user receives in the answer only the dimensions available to them; otherwise, the assistant becomes a workaround channel to restricted data.
- Answer quality validation: we prepare a set of questions for each metric, compare numbers with the reference, separately test ambiguous wording and queries outside the data domain. A correct “there is no data to answer” is more valuable here than a plausible guess.
- Refinement based on test run results: we clarify metric descriptions, add missing fields, adjust dialogue prompts until the share of correct answers stabilizes.
- Team training and handover of procedures: a walkthrough with analysts and metric owners, instructions — how to add a metric, how to report an error in an answer, who is responsible for keeping descriptions up to date.
At each stage an artifact remains: a source map, metric descriptions, an access matrix, a validation protocol, a support procedure. Acceptance is based on them — and the team works from them after launch without our involvement.
Artifacts and documentation that remain with the team
Work on BI AI Copilot does not end with service launch: the team receives a set of documents by which the solution can be developed, handed over to new engineers, and defended on the customer side. Below is what remains in the project files after delivery.
- Data relationship schema — which sources are connected, how entities are joined, and where each metric is calculated; from it, a new engineer restores the logic without reading query code.
- Metric registry — name, formula, source, owner, and validity period for each metric; the dispute over “whose number is more correct” is closed with a reference to a specific row.
- Access policies — roles, data dimensions, and export restrictions are defined at the model level rather than in an individual report, so they do not diverge between dashboards.
- Discrepancy investigation procedure — a step-by-step order of actions when the number in a report does not match the source: reconcile the load, find the cause, record the conclusion, and correct the calculation rule.
- User instructions — how to phrase a question to the data, how to verify the answer received, and what to do if the model answers uncertainly.
- Data quality check map — control of completeness, duplicates, and period gaps at the input; when a check triggers, the report is marked as incomplete.
- Change log of metrics and calculation logic — version, author, and reason for the change; reverting to a previous calculation takes minutes, not days.
Case: from manual exports to natural language questions
Starting point
A distributed team: analysts in two time zones, the product manager in a third. Metrics lived in different environments — summary tables, reporting in Power BI, manual exports from billing.
The answer to “why conversion dropped” took two or three days to assemble: the analyst wrote a query, checked definitions, and rebuilt the breakdown.
In meetings, up to four discrepancies per meeting: marketing and finance calculated “active customer” differently, and half the time went to determining whose report was correct.
What was changed and what was achieved
We assembled a unified semantic layer — metric definitions, granularity, filtering rules, and duplicate exclusion in one place, with versioning alongside the models.
On top of it, we set up BI AI Copilot: the question is asked in natural language, and the answer arrives as a ready breakdown with a reference to the metric definition. Rights are inherited from the catalog, so financial metrics do not leak into the general chat.
Answer preparation time is minutes instead of days. The share of manual exports is about one tenth: nonstandard investigations remain, where a causal model is needed rather than a breakdown. Disputed numbers are caught before the meeting, at the input to the model.
New metrics are described once — then they are available both in the report and in the assistant’s answer, so the discrepancy between the “analytical” and “business” number disappears as a class.
How the semantic layer differs from model query generation
The boundary runs along who defines the calculation logic. The semantic layer is pre-described metrics, relationships between tables, aggregation rules, and filters agreed with the business.
The model does not reinvent the formula: it selects a suitable metric from those described and substitutes the needed dimensions.
Query generation solves a different task — it translates human-language wording into a specific query to the data. If the definitions lie in the same layer, the query is assembled from ready-made building blocks: metric, period, dimension, comparison with the previous period. The model is responsible for understanding the question, not for arithmetic.
Without the layer, the same question yields different numbers: revenue is calculated sometimes with tax, sometimes without, by shipment date or by payment date. The model chooses a plausible option each time, and the previous answer can no longer be repeated.
The semantic layer fixes one definition of a metric, and its change goes through review rather than appearing unnoticed in a new query.
Hence the reproducibility of the answer. The same question asked a day or a month later relies on the same logic, so the number can be reconciled with the report and the discrepancy explained.
The log shows which metric was selected, which filters were applied, and which version of the definition the answer was built on.
Practical consequence for BI AI Copilot: a new metric is described once — in the layer — and is immediately available to all question wordings. The model remains an interface to agreed data, not a source of truth.
Common concerns about security and answer quality
Concerns before connecting BI AI Copilot to a production environment concern not the model but access to data and trust in the number. We break down four areas that most often require explanation.
How is access to data from different departments separated?
Rights are inherited from the account in the corporate system: an employee sees only the dimensions allowed by their role. Filters are applied on the source side, before data reaches the model, so the answer cannot show what there is no access to.
What to do if reference data is incomplete or inconsistent?
Incompleteness is a normal state of data, and it is handled explicitly. Metrics without sufficient attributes are marked as limited reliability rather than silently estimated.
Discrepancies between systems are displayed as a separate block so the metric owner can see the source of the conflict.
Who is responsible for an incorrect number in the answer?
Every number is traceable to the source: it is visible from which table, by which formula, and as of which date it was taken. In case of a discrepancy, it is not the answer that is corrected but the metric definition — after that, all reports are recalculated in the same way.
How difficult is it to connect a new source?
A new source is an integration task, not a matter of rewriting question logic: entities, relationships, and aggregation rules are described. For typical accounting systems this takes days of work, for custom databases longer — first we fix the metric definitions.
Let's discuss the task: what to send for an estimate
To come back with an estimate and a work plan rather than clarifying questions, we need enough input on your environment. Below is what to send: this is enough to estimate the scope, risks, and sequence of BI AI Copilot implementation stages.
- Data sources. A list of tables, data marts, reports, and dashboards that the assistant must cover, plus a schema of relationships between them. The exports themselves — later.
- Access and environment. Your server or cloud, a secure channel, a read-only account, whether there is an intermediate layer between the warehouse and analytics.
- Key metrics. Metrics with calculation formula and granularity: day, week, region, channel. We fix disputed definitions at the start; otherwise, the assistant will answer in its own way.
- Glossary of terms. How the company refers to customers, revenue, active contracts — the assistant must answer in the department’s language, not the language of tables.
- Example questions. 10–15 real requests from managers and analysts: we use them to check that the assistant understands the wording and does not invent numbers.
- Access rules. Masking of personal fields, segregation by department, query log, and history retention period.
- Update procedure. Load frequency, nightly recalculations, behavior during a load failure — this determines which assistant answers can be considered current.
- Reference point. One report where you know the correct numbers for certain. It becomes the benchmark for validating the assistant’s answers.
This set is enough to come back with an estimate, a stage plan, and a list of risks for your environment. Describe the task — we will gather the inputs into one list and propose the order of work.
