AI data analysis: where the manual loop breaks
AI data analysis rests not on the model, but on the manual loop around it.
A business question arrives phrased in natural language, an analyst translates it into a query, agrees on the sampling logic with the metric owner, waits for the export — and goes through this path again for each next phrasing.
The bottleneck in analytics here is not query execution speed, but the approval cycle.
As long as sampling logic is not fixed and reproducible, any calculation remains a draft: the number cannot be defended before a manager, and manual reporting exports outlive the hypothesis they were made for.
SQL generation from a question shifts the analyst's work from typing to checking meaning.
The model builds a draft query from the description and data schema, while the human controls what actually breaks reports: period boundaries, table joins, duplicates and gaps, metric definition.
Each such query becomes reusable: identical questions get identical logic, not nine different interpretations of “active customers.” The queue changes too — some questions are resolved directly from the interface, without reaching an analyst.
Next we examine exactly where the manual loop fails and how it is closed: step by step, with the result checked against historical data and already-approved reports.
The effect of Text-to-Code in data analysis
Text-to-Code does not replace the analyst, but takes routine work off their plate. The model parses the natural-language phrasing, checks it against the data mart schema, and assembles SQL or Python that goes straight to execution.
The difference is visible not in a demo, but in the task flow: routine requests stop waiting in the queue.
How much time typical query generation saves
A typical query is a join of several tables, a period filter, and aggregation across two or three dimensions. Manually, such a query takes from half an hour to two hours, and if fields and granularity need to be clarified with the client, longer.
Generation produces a working draft in seconds, and a person checks only the meaning of the result: for our clients, 55–70% of such requests are resolved without an analyst.
The class of errors that goes away with manual code
The main source of errors in manual code is join logic: incorrectly chosen join type, lost rows with an inner join instead of an outer join, different key granularity, duplicates on many-to-many relationships.
The generator builds a join graph from the relationships described in the data catalog, so this class of errors does not reproduce; the result is reconciled against control totals, not by eye.
“Every generated query passes a permission check at the role level: if a user does not have access to the data mart, it will not execute, even when the code is assembled correctly,” — an engineer on our team.
Together, this redistributes the workload: analysts move away from a stream of repetitive requests toward modeling and investigating disputed data, and the response time for a typical question is measured in minutes instead of days.
Query generation modes: comparing options
The generation mode is chosen not by the “power” of the model, but by data sensitivity and the cost of an error in the query. Below are the real alternatives among which the choice is made, and what each is limited by in practice.
| Option | When it fits | Limitations |
|---|---|---|
| SQL generation only | Data marts and reports on a stable schema | The model will not handle multi-step logic or go beyond the syntax of a single dialect |
| SQL and processing script generation | Data preparation is needed before calculation | An error in an intermediate step is harder to localize |
| Mandatory schema and permission check | Access to sensitive tables | Some queries are filtered out; an up-to-date metadata catalog is needed |
| Local model in a closed perimeter | Data does not leave the client's perimeter | Smaller context window, weaker on non-standard phrasings |
| Hybrid: generation + manual review | Statutory reporting and recurring calculations | Review adds delay; an engineer is needed in the loop |
Where the model's applicability boundary lies
The boundary lies where schema description ends and domain knowledge begins: relationships between reference tables, historical tracking, rules for excluding duplicates.
The more complete the metadata catalog and role permissions given to the model, the more accurate the generation — but we still leave data schema validation as a mandatory step, not an option.
Limitations on load and data volume
The model works with the data description, not the rows themselves, so table volume has almost no effect on query quality — the number of relationships and ambiguity of names do.
We enable hybrid mode when a query becomes part of recurring calculation: generation assembles a draft, and review locks in the verified version.
Implementation stages: from source audit to handover
We start not with the model, but with data inventory: as long as it is unknown where metrics live and who is responsible for their correctness, the generator produces convincing but incorrect numbers.
- Data source audit. We record where each metric comes from, where data diverges, and which tables duplicate each other.
- Glossary of terms. We agree on the wording of metrics and dimensions, and map each term to tables and a calculation formula.
- Accesses and roles. We open read-only connections and scope visibility by department.
- Connection to the warehouse. We load the schema, glossary, and calculation rules, and test the setup on live questions.
- Benchmark questions. We assemble a control set of typical questions with known answers in advance.
- Accuracy measurement. We compare answers with the reference, analyze discrepancies, and refine the glossary up to the target share of correct answers.
- Solution handover. We provide instructions, the glossary update policy, and train the team.
How we describe business terms for the model
A term is described in three parts: what the business calls it, which tables the data comes from, and what formula calculates the metric. We tie department synonyms to the same term — the questions “revenue by region” and “sales in the area” lead to the same calculation.
How we validate accuracy on a control set of questions
We assemble the set from real client questions with manually calculated answers. After each glossary edit, the run is repeated — it shows that the refinement did not break previously correct queries.
What we hand over to the client along with the system
The system cannot be considered delivered until the client's team has documents by which it can operate without us. Therefore, along with the working environment, we hand over a package describing not only what to click, but also why the solution is designed this way.
- Source and relationship schema — which tables, exports, and reference tables participate in the analysis, by which keys they are joined, and where discrepancies are possible.
- Access policies and roles — who sees raw data, who works only with aggregates, who has the right to change query generation rules.
- Control question set — typical analyst phrasings with the reference result: they are used to check the system after each update.
- Instructions for analysts — how to phrase a question to get a correct query, and what to do when there is a discrepancy with expectations.
- Saved query catalog — verified SQL and Python scripts with notes on which can be reused.
Glossary of terms and translation rules
A key document: it records how business words become database fields. “Revenue” — is it shipment or payment, for what period, with what exclusions. Without a glossary, two analysts ask the same question and get different numbers.
Support policy when the schema changes
A renamed field or a new order status breaks some queries.
The policy describes who reports a change, within what time frame the glossary and translation rules are updated, and how control questions are run — a discrepancy is identified before the analyst sees an incorrect number.
Case study: eliminated manual exports and sped up queries
What it was like before implementation
The business unit collected operational reporting manually: about 45 requests a week went to analysts, and the queue built up for 2–3 days.
Each report was assembled from exports from several systems, and the same metric in different files was calculated differently — the share of discrepancies reached 15%. Decisions were made on data that had time to become outdated while the report was being prepared.
We connected a Text-to-Code layer on top of the existing warehouse: an employee phrases a question in words, the system builds a query against approved data marts and returns the result in seconds.
Access rights, reference tables, and metric definitions remained the same — only the way to get a query changed.
The result in numbers and what changed in the process
- 68% of requests are resolved independently, without an analyst: a report “in the moment” instead of waiting in a queue.
- Report waiting time dropped from 2–3 days to a minute, and recalculation happens immediately after the phrasing is refined.
- The share of calculation errors fell from 15% to 3%: answers are built on a single metric semantics, and manual copying is gone.
- The flow of requests to analysts decreased from 45 to 13 per week — the freed-up time went to modeling and data quality.
“Previously, half of planning meetings began with a question about why the numbers didn’t match. Now we argue about decisions, not exports,” — Head of Analytics at Nordline Retail.
How does query generation differ from report dashboards?
A ready-made report is a predefined question to the data: metrics, dimensions, and filters defined before the user opened the dashboard. It answers quickly and identically for everyone because the calculation relies on a described model and is cached.
But as soon as a non-standard dimension is needed — a different combination of measures, a new cohort — the ready-made set ends, and the request goes into the developer's queue.
Generation of arbitrary queries begins where a question appears faster than it can be described in a report. The analyst phrases it in words, the system assembles SQL or a Python script against the actual data schema, checks table and field names, and returns the result.
What matters here is not “language magic,” but two constraints: construction proceeds only against the described schema and with read permissions, and the query itself remains verifiable — it can be shown, corrected, and saved.
One does not replace the other: a dashboard is predictable, cheap to calculate, and suitable for regular metrics where everyone looks at the same numbers.
On-demand analytics wins in flexibility, but requires resources for execution and validation of the result — without it, it is easy to get a syntactically correct and meaningless answer. Therefore, it is reasonable to keep both layers together, separating them by purpose.
In practice, regular metrics live in ready-made reports, while generation works on top of the same semantic model — shared definitions of fields, metrics, and filters.
Then an arbitrary query does not diverge in numbers from the dashboard, and a successful query becomes a candidate for a new report. The difference between the approaches is not in which is more modern, but in how well the question was known in advance.
Technical concerns before project start
Technical concerns usually surface at the estimation stage, not during the pilot — and that is the right moment to resolve them. Below is what engineers and business unit heads most often ask before the start.
How accurately does the model translate a question into a query?
Translation accuracy is checked not in a demo, but on your set of questions: we collect 50–100 real analyst phrasings, run them, and compare the result with reference queries.
The model returns not only the query but also an intermediate representation — it shows where it guessed the condition and where it inserted an incorrect filter.
The system reads the table schema and relationships from the warehouse metadata, so it relies on actual field names, not guesses.
Will heavy queries bring down the production warehouse?
Queries go to a separate execution environment with limits on time and output volume, not directly to the production database. The system rewrites aggregates, applies date partitioning, and cuts unnecessary joins — on typical reports this reduces response time severalfold.
If the estimated query cost exceeds the threshold, it does not run: the user receives a warning and a suggestion to narrow the period.
How do you separate access between departments?
Permissions are inherited from your authentication system, and visibility rules are described at the table and row level — an employee of one business unit will not see another's data, even if they phrase the question correctly.
Every model request and every generated query goes into a log indicating the user, time, and affected entities.
Can it work without external services?
Yes. In a closed perimeter, a proprietary translation-to-query model is deployed, without calls to external APIs — data does not leave your perimeter. We calculate hardware requirements in advance based on the volume of metadata and the number of concurrent users.
What about an already deployed warehouse?
The existing warehouse remains the source of truth: we do not migrate data or change the schema, but connect to it as an external source.
The description layer — which tables and fields are available to the model, what business entities are called — is configured on top and updated as the schema evolves.
Let's discuss the task and come back with an estimate and plan
To make the conversation about an AI data analysis system move straight into estimation rather than general discussion of the technology's capabilities, it helps to come with concrete facts.
Below is what actually affects architecture and timelines: based on these points, we calculate effort, choose the method of connecting to warehouses, and define the boundaries of validation.
- Data source list: which databases, warehouses, and files are in use, how often the schema changes, and whether there are data marts with already-approved metrics.
- Examples of typical questions: 15–30 phrasings from analysts and managers — query generation is tuned on them, and accuracy at acceptance is measured by them as well.
- Access requirements: a read-only account, permitted schemas, where the system may write the result, and what is prohibited from being read and stored in logs.
- Perimeter constraints: where the system is deployed, whether there is an isolated segment, and your own rules for storing query history and anonymizing data.
- Success criteria: what we consider success — the share of questions resolved without an engineer, time to first answer, and the share of calculations with mandatory human review.
- Owners and acceptance: who is responsible for metrics and who accepts the stages of the implementation plan.
Describe the task in free form — we will come back with a project estimate and implementation plan, where each stage has a fixed expected result and checkpoint.
