Cost tracking AI: why token consumption grows unnoticed
Token consumption almost never grows linearly with the number of requests.
The load is the same, but consumption has increased severalfold: the prompt got heavier, dialogue history, search results and tool descriptions were added to the context, part of the traffic moved to a more expensive model.
The request counter does not show this — it counts calls, but we pay for tokens, and these two curves diverge more and more over time.
A separate problem is cost attribution. While model request accounting is done as a single total per key, it is impossible to say which product, scenario or operation type consumed the budget: a chat assistant, background document parsing or nightly embedding recalculation.
Without breakdown by models, clients and call type, decisions are made blindly — they optimize what sounds expensive, not what actually drives consumption.
Meanwhile, call limits are set by eye: based on average consumption for the previous period, without a margin for peaks and retries.
Retries, agent looping, a one-time import of a large corpus — and the threshold is breached, while consumption anomalies are noticed only after the fact, when the numbers have already accumulated over weeks.
Below we look at how this is closed: telemetry for every call, attribution by product and model, thresholds and alerts before consumption goes up.
What end-to-end tracking of model request consumption provides
When all model calls go through a single gateway, each call is tagged with product, team and scenario.
The share of untracked calls drops to zero: the report shows not total consumption but the contribution of a specific service, and the discrepancy with the provider's data stays within fractions of a percent.
Cost attribution by products and teams
The breakdown shows that the main volume is driven by one or two scenarios out of ten, while the rest consume single-digit percentages.
The team sees its own area and fixes it itself: rewrites the request template, shortens the dialogue history summary, moves simple tasks to a compact model.
After such a fix, the share of unnecessary calls drops 2-4 times — the cause is visible in the breakdown, not in the total bill.
Limits and consumption predictability
A threshold per request, scenario and day triggers before consumption goes beyond the allotted band. Looping, retries during a connection drop and a spike from unfiltered input hit the limit and go to a queue instead of eating up the reserve overnight. The monthly volume deviates from the plan by percentages, without emergency expansion.
Tags reduce incident investigation time: a spike is traced through the log in minutes, not in a day of correspondence between teams. At the same time, a class of failures goes away — silent quota overruns at the provider, duplicate retries and calls from a decommissioned service.
Ways to track AI requests: gateway, instrumentation, logs
Cost tracking of requests — cost tracking — is built on three levels: a common gateway in front of providers, instrumentation inside the application, and post-hoc analysis of provider logs.
They differ in attribution accuracy and maintenance cost, so in practice they more often converge to a hybrid scheme.
| Option | When it fits | Limitations |
|---|---|---|
| Tracking gateway | Several teams and services access models through a common point | Sees only calls that pass through it |
| Application instrumentation | Need breakdown by product, team or scenario | Changes in every service and maintenance during updates |
| Provider log export | Need reconciliation of actual charges | Arrives with delay, without breakdown by consumers |
| Hybrid scheme | Need both attribution and reconciliation | Two data models and rules for reconciling them |
Attribution accuracy
The gateway records every call, including retries and streaming responses, and gives full input and output consumption — but it does not know which product is behind the key, so attribution is passed in request headers.
Instrumentation accurately names the consumer, but only where the code is actually instrumented. Provider logs serve as external reconciliation: they show actual charges and discrepancies with internal accounting.
Overhead in production
The gateway adds latency to every call and becomes a critical point — its failure stops model calls. Instrumentation does not affect response speed, but requires discipline in teams.
Log export does not load the system at all, but it lags, and you cannot limit consumption in the moment based on it.
How we implement token consumption monitoring: work stages
Cost tracking of AI requests starts not with a dashboard but with an inventory: who accesses models and with which keys. The work order below gives cost breakdown by team, product and scenario, not a single sum at the end of the month.
- Audit of calls and sources. We compile a registry of services, keys and models; separately mark background tasks, retries and streaming responses — they most often produce untracked consumption.
- Tagging scheme. We define mandatory labels: team, product, environment, scenario. A call without a label does not pass gateway validation, otherwise the share of unidentified consumption grows.
- Setting up metric collection. The gateway in front of the model records input and output tokens, model type, duration and stream flag; aggregates go to storage with a time step.
- Check on streaming and repeated calls. With streaming output, the counter arrives at the end of the response; with a retry, it is duplicated: here we catch double counting and losses during a connection drop.
- Reconciliation with the provider report. We compare aggregates for identical periods: a discrepancy within a couple of percent is rounding, a larger one is analyzed by specific calls.
- Handover of procedures. We provide the label description, alert thresholds, anomaly investigation procedure and rules for adding new scenarios.
Checking convergence with provider data
Reconciliation is ongoing, not once a quarter: we compare daily aggregates with the issued invoice and break down discrepancies by specific calls.
This surfaces incorrect accounting of cached responses, losses during a connection drop and duplicates on retry — after that, the dashboard figure matches the invoice, and it can be relied on for planning.
What we hand over: configurations, policies, procedures
We hand over not a presentation but a working environment: configurations, policies and procedures live in the customer's repository, are versioned and tested on a staging bench before deployment.
Each artifact describes not "what is configured" but why exactly that was chosen — so the environment can be changed without us.
- Gateway configuration: request routing rules, limits per project and team, input and output token counting, degradation mode when quota is exhausted.
- Tag and attribution scheme: naming convention (project, team, environment, task type) and linking consumption to the owner — without this, consumption cannot be broken down by consumers.
- Consumption dashboards: summary by tokens and number of requests, dynamics by day, top consumers, share of repeated calls and cache hits.
- Access policies and roles: permission matrix, separate keys per service, limit change log.
- Escalation procedure: to whom and within what time the deviation signal goes, who makes the decision to restrict.
- Documentation and knowledge transfer: flow diagrams, field descriptions, instructions for editing limits, analysis of two training incidents.
Access policies and roles
The role matrix separates those who see aggregated metrics from those who change limits and routes. Keys are issued per service, not "one for all": if compromised, one access is revoked, the rest continue to work.
The role model is linked to the customer's authentication system, so when an employee leaves, their access to consumption data is automatically closed.
Procedure for responding to anomalies
The threshold is set not by an absolute number but by deviation from the baseline for previous periods — this way the rule does not trigger on planned growth.
The signal comes with the project, team and task type, and the decision to restrict is recorded in the log with the reason. After a month of operation, thresholds are revised based on accumulated statistics.
Case: how token consumption was cut threefold without quality loss
The customer is a platform with several product teams, each accessing models directly from its own services. The load grew moderately, but token consumption grew faster: there were no consolidated figures for all callers, and part of the traffic did not get into reporting at all.
What pre-implementation measurements showed
We collected scattered logs for a month and consolidated them by unified attributes: model, team, environment, call purpose, retry flag.
The picture turned out unpleasant — 23% of calls were not tracked anywhere: background tasks, automatic retries after errors, test environments and bypass paths around the common gateway.
The spread of consumption between teams with comparable load differed fourfold: part of the difference was explained by prompt length and lack of caching, part by duplicate calls of the same task.
The changes came down to three things: a single gateway through which all calls go, attribution of each request with team and purpose tags, and a limit with an alert when deviating from the baseline over the observation window.
We collect attributes in a unified format — OpenTelemetry — so metrics are consolidated between services without manual exports.
Two months after implementation: the share of untracked calls — 1%, the spread of consumption between teams decreased from fourfold to 1.3, anomaly response time — from a day to 15 minutes.
Total token consumption fell threefold, while quality scores on the reference task set remained at the same level.
How does tracking in a gateway differ from code instrumentation?
The gateway counts tokens where it sees traffic as a whole: it intercepts the request and response and takes actual values from the service block of the response — system prompt, dialogue history, tool calls.
Accuracy is high, but only for the environment that goes through it; the price is an extra network hop and a single point of failure.
Code instrumentation is arranged differently: measurement is placed around the model call inside the application. This shows business context — user, scenario, logic branch — but only what the developer explicitly wrapped is counted.
Dynamically assembled prompts, background tasks and calls from auxiliary services remain outside accounting.
Both approaches lose accuracy on the same things. With streaming output, final values arrive with the last fragment and are lost if the connection drops earlier.
Retries after an error or switching to a backup model will be counted by the gateway as separate requests, but by the application as one; the count diverges in both directions.
The main source of discrepancies is bypassing the gateway: direct keys in laptops, scripts and internal services are visible neither to the gateway nor to instrumented code.
The hybrid closes this: the gateway remains the source of truth for tokens, while the application passes scenario and user labels through headers, and the gateway metric is broken down by product slices.
If some calls already go outside the controlled environment, we start with an inventory of exit points and parallel counting: the gateway and code count the same scenarios, and we analyze the discrepancy by groups.
Then we migrate traffic in parts, and close what remains outside the gateway with instrumentation, reconciling both sources in a common reconciliation.
Frequently asked questions about model call tracking and limits
Model call tracking is designed so as not to fall into the hot response path: consumption is captured by the call gateway asynchronously, after the result is returned, and written in batches. Therefore response time is determined by the model itself, not by telemetry.
The contentious points are different — streaming output, retries and calls going past the common gateway. Let's break down each.
Does the token counter add latency to the response?
No, if tracking is moved to the gateway and does not block the stream. Tokens are counted from the already received response — either from the final stream event or by a local tokenizer, and writing to the metrics storage goes through a separate queue.
The client receives the last chunk at exactly the same time as without tracking; latency is possible only with synchronous database writes for each request — that is not in the scheme.
How to separate consumption if several products go through a common call service?
Each consumer is issued its own key to the gateway, and a product and calling service label is added to the request metadata. Aggregation is built across three dimensions: access key, endpoint, operation type.
Then the common service remains a single entry point, but the count for each direction converges separately, without manual cost allocation.
What happens with streaming responses?
Consumption is taken from the final stream event, where the provider gives final statistics.
If the provider does not send it, tokenization is counted incrementally on the gateway side; the discrepancy with provider billing remains within a few percent and is visible in the reconciliation report.
Are retries and timeouts counted?
Yes. A retry after a timeout is a new actual call, and it is billed by the provider.
The gateway writes each attempt as a separate record, while an idempotency key links them into one logical operation: both the final cost of the task and the cost of instability of a specific scenario are visible.
Can tracking be bypassed?
Only by going past the gateway — directly calling the model from code with a provider key.
This is closed at the access level: provider keys do not get into applications, outbound connections are restricted, and a discrepancy between the number of operations in the application and in tracking raises an alert.
Let's discuss the task: we will return with an estimate and tracking plan
To make the estimate not "by eye" but to the point, we need source data on how model calls are currently arranged. We collect it in one place — from the list of models to those responsible for access.
Then we calculate the cost of tokens for each route and determine control points in advance, where deviation is visible before it becomes a surprise in reporting.
What to prepare for the conversation:
- Problem statement: what result is needed — a one-time consumption snapshot, a regular report by teams or a rule that stops requests on overspend.
- Call scheme: which services access models, through which gateway, where routes are duplicated and which of them are bypass.
- List of models and sources: list of models, context handling modes, who exactly sends requests — a service, an operator or batch processing.
- Service scheme: map of components and queues, to link tokens to a specific product or team, not to a common counter.
- Current limits: quotas, rate limits, everything already configured on the model provider side.
- Access owners: who issues keys, who approves configuration changes and where the notification arrives.
- Control points: where the report reaches people and who responds to deviation from the usual consumption profile.
Based on this data, we describe the implementation plan by stages — with verification at each, not one big switch. Describe the task and send what you already have: we will return with an estimate and tracking plan.
