Token counting: where request spend gets out of control
Any request to a language model is paid not for the meaning of the answer, but for the volume of context: token counting is based on the input instruction, dialogue history, and generated response.
While there is only one scenario and the conversation is short, request spend is predictable and within expectations. As soon as long dialogue context, attached documents, and call chains appear, the counter on the service side stops matching what the team planned.
The first leakage mechanic is history accumulation. Each subsequent turn in the dialogue resends the entire previous conversation along with the system instruction, so the tenth message costs many times more than the first, even though the user wrote one line. Without token counting for each step, this becomes noticeable only at the end of the period.
The second is repeated model calls. Timeout retries, fan-out parallel requests, re-requests for every action in the interface: the same fragment of context is paid for several times. This also includes unchanged system blocks that are sent at every step and are not cached.
The third is the lack of accounting by users and scenarios. When request spend is a single total, you cannot answer which function or which account takes the main share. Breakdown by scenarios, clients, and operation type turns the bill into a manageable value.
We examine how this is closed: where to place counters, how to trim context without losing quality, and how to set limits before spend goes up.
What token counting provides: control, limits, reporting
Each call to a language model is counted in tokens: input — the prompt together with dialogue history, output — the response. For OpenAI and other model providers, pricing is built exactly on this, so unaccounted spend is a derivative of who asked and how often.
We capture the counter on every call and tie it to the user, service, and session — then it is no longer a raw log but a metric.
Spend accounting broken down by users shows where the main volume goes: one department contributes heavy long prompts, while frequent short reference queries cost almost nothing.
Separately, the cache hit rate is visible: repeated context is not paid for twice, and this is the fastest lever for reducing spend.
Anomalies — a request with a history of tens of thousands of tokens, cyclic retries, a forgotten debug script — are found in the first days, not at the end of the period.
Request limits are set separately by number of calls and by token volume per minute — per user, team, and service. A soft threshold routes the request to a queue or shortens the response, a hard one returns a refusal; the choice depends on the scenario. Peak loads are smoothed with a queue, batch processing, and cache — spend stops growing in steps.
Reporting is a daily slice by services and users: token volume, dynamics, source of the spike. It shows what to change: prompt wording, length of transmitted context, or the model itself.
Accounting modes: proxy layer, call wrapper, provider logs
Spend can be counted at three points: at a network intermediary, inside application code, or from the model provider's logs.
The points differ in what each sees: the intermediary sees only the request and response body, the wrapper also sees the call context, and logs contain already counted values.
| Mode | When it fits | Limitations |
|---|---|---|
| Proxy layer | Need unified control over all services at once, including third-party applications, without code changes | Terminates the connection and requires trust in the node; does not know logical boundaries of the operation — one task is split into several calls |
| Call wrapper | Accounting needed by operations and users, with limits and spend estimation before sending | Works only where it is connected; direct calls bypassing the wrapper fall out of accounting |
| Provider logs | Reconciliation and control: figures come from the provider, no discrepancies with billing | Delay from minutes to hours; no breakdown by internal operations; some calls may be unlabeled |
Response caching adds a fork.
A cache hit does not consume tokens, but whether to charge it against the user's limit is a policy question: when charging on the fact of the request, the cache quietly eats the budget; with a complete refusal to charge, there is an incentive to split identical requests.
It is more practical to combine modes: the wrapper provides an operational limit and pre-send estimation, the proxy covers services without code changes, and logs serve as the source of truth for reconciliation. A discrepancy between them is a signal that some traffic bypasses accounting.
Implementing spend accounting: from call audit to policy
The work order is the same for a new service and for an existing one: first we measure, then we limit. We start with a call audit — it shows where spend is actually generated, not where it is expected. Then the accounting scheme, limits, load testing, and handover together with the policy.
- Call audit. Inventory of flows: who initiates the request, how much context goes to the model, which calls are duplicated, where a full response is generated for a single line. The result is a map with each scenario's share in the total volume.
- Accounting scheme. We choose the measurement point: application before sending or proxy layer. We fix what we attribute to a request — context, dialogue history, model response, tool calls.
- Token counting. One tokenizer for the target model, otherwise the report diverges from the provider's bill. We count input and output separately, with a breakdown by scenarios.
- Limit configuration. Three levels: daily per scenario, per user session, and an emergency stop for the entire service. When exhausted — predictable degradation instead of silent overage.
- Load testing. Profile of real requests: how counters behave under parallel requests and connection drops, whether streaming responses are included in accounting. A discrepancy with provider data of a few percent is normal; more means an error in the measurement point.
- Policy and handover. We provide a monitoring dashboard, rules for revising limits, the procedure for investigating discrepancies, and the owner of each scenario. Then prompt edits follow the same loop: change → measurement → limit.
Handover package: accounting schemes, limit policies, regulations
Implementation ends not with a working counter, but with the moment when the client's team changes limits, investigates overages, and connects a new service without us.
Therefore, we hand over not a report on work done, but a working package: configurations, schemes, and regulations that live in the same code repository and are updated together with the service.
- Data flow scheme: at which point token spend is counted, where aggregates are collected, how they reach reporting, where the failure points are, and who is responsible for each section.
- Limit policies: thresholds per division, service, and individual user plus behavior when exhausted — warning, refusal, or switch to a more economical model.
- Escalation regulations: at what deviation from the usual spend profile an alert is raised, who responds and within what time frame, who has the authority to temporarily raise the threshold.
- Handover documentation: deployment procedure, updating the rate reference, connecting a new model, and checking that accounting has not diverged from consumption.
- Reconciliation scenarios: a set of requests with known spend that is run after every change and catches divergence before it reaches reporting.
- Reporting and exports: spend by teams and scenarios, share of repeated requests closed without calling the model, dynamics over an arbitrary period.
The package remains functional without us: policy edits go through normal review, and a new engineer enters the loop via documentation, not colleagues' stories.
Case study: language model spend stopped growing while load increased
What was
A first-level assistant conducted dialogues in chat and email: conversation history, knowledge base excerpts, and system instructions went to the model in one block. Requests grew threefold over the season, and model spend repeated this graph almost linearly.
Spikes added their own: when the provider limit was exhausted, the request failed with an error and went to retry, and the retry was billed as new.
What was changed
Token accounting was moved to a gateway in front of the model — input and output were counted separately, with a breakdown by scenario, user, and model class; metrics were captured via Langfuse.
From them it became clear that 60% of the volume is consumed by three scenarios out of twelve, and the model does not use more than half of the transmitted dialogue history.
Next — limits instead of refusals: truncating history to the last messages while keeping a brief summary, routing short requests to a junior-class model, caching repeated prompts within the dialogue window.
For each scenario, a threshold was set: a warning at 80% and smooth degradation instead of an error when exhausted.
What became
| Metric | Before | After |
|---|---|---|
| Requests per month | baseline level | three times more |
| Tokens per request | full context | three times less |
| Share of retries after error | 18% | 4% |
| Refusals due to provider limit | regular spikes | zero |
Spend stopped following load: there are three times more requests, token volume grew by only a fifth and holds at that level. A second effect appeared as well — the breakdown shows in advance how many times a new scenario will increase load, even before rollout to production.
What to do if limits are already exceeded and spend continues to grow?
Exceeding limits is almost never related to rate increases — token consumption grows in one specific loop: bloated context, a loop in the agent, retries after failures, or a background task without a cap.
Therefore, the order of actions is counterintuitive: first stop the growth, then look for the cause. Diagnostics on live traffic that continues to consume quota will show nothing.
The first step is to cut low-priority traffic: batch processing, reindexing, background enrichments. This removes a noticeable share of spend within minutes and does not affect user scenarios. Then spend is broken down along axes: access key, model, method, tenant, hour of day.
Provider request logs — OpenAI and analogues provide such a breakdown — are compared with last week's baseline and we find the point where the graph went up.
Emergency thresholds are set in advance: 70–80% of the daily quota — notification to the responsible person, 100% — automatic switching of low-priority scenarios to a more economical model or refusal of service.
After stabilization, thresholds are revised according to the actual load profile, otherwise the protection will trigger on a normal peak.
Common technical doubts about spend accounting and limits
Doubts about accounting usually concern not the formula itself, but the counter's behavior in real traffic: does it reconcile with the issued bill, what happens on a streaming response, how much the control itself costs. We address the questions engineering teams ask most often.
How accurate is token counting and where does it diverge from the provider?
We count with the same tokenizer as the target model, so divergence occurs only on service wrappers and system messages. In the report we store both values — ours and the one confirmed by the provider's response — and show the difference in a separate column.
If the divergence exceeds the set threshold, it is a signal that the model was updated and the markup needs to be rechecked.
How are tokens counted during streaming output, when the response arrives in fragments?
The counter incrementally accumulates fragments as they arrive and records the total at the moment the stream ends — we do not write intermediate values to history, otherwise we would get dozens of false records per call.
For dropped connections, we save an estimate based on the prompt and the text that has already arrived, with a separate flag, so that such records are not mixed with complete ones.
How much does counting slow down the response — does latency increase?
The control layer sits on the request path and adds single-digit percentages to the total response time, not a fixed amount. With streaming output, the first fragment goes to the client before the entire response is counted, so the user does not notice a difference.
If the layer becomes a bottleneck, we move it out of the main path and count asynchronously, upon call completion.
What exactly is stored in the call history and how does it not grow indefinitely?
We write compact records: request ID, model, counters, stream flag, timestamp, and limit check result.
Prompt and response texts are stored optionally, with a configurable retention period and scheduled cleanup — then the history remains suitable for incident investigation, but the volume does not grow indefinitely.
Let's discuss the task: how to bring spend accounting under control
So that the implementation estimate is not "by eye" but based on facts, we need input data on how requests to models are currently consumed.
Below is a short list: if some items are missing, that is also an answer — it shows the amount of preparatory work in the work plan.
- list of services and scenarios that call models, with approximate call frequency;
- anonymized samples of requests and responses — they show where context grows without benefit to the result;
- current limit rules and how overages are currently handled;
- export of request logs for a typical period: a calm week and a peak load;
- routing scheme: which models and in which modes are used, where there is headroom for switching;
- reporting requirements — in what dimension spend needs to be seen: by product, team, or client;
- infrastructure and access constraints, if accounting needs to be embedded into the existing loop.
With this set, we come back with an implementation estimate and work plan: sequence of steps, control points, what and when will appear in reports. Describe the task in a convenient form — we will discuss what will give an effect immediately in your case, and what can be skipped.
