ONNX Runtime inference optimization: where speed is lost
Model response latency is the sum of stages, not the time of a single run. A request goes through input preparation, computation, post-processing, and waiting for a free resource, and at each step time is lost for its own reason.
When only the run itself is measured, the picture looks brisk, but the user still waits — the losses are hidden at the junctions.
The first area is input preparation: tokenization, type casting, tensor layout, and copying data into the compute device's memory.
On short requests and compact models, these operations become the bottleneck of request processing: layers are computed faster than the data can reach them. A model execution profile captured by stages rather than as a whole immediately shows this asymmetry.
The second area is the run itself and the load on the compute device. Here time is lost on suboptimal kernels, unnecessary format conversions, and a graph that was not built for a specific accelerator.
ONNX Runtime provides a set of levers for this: an execution provider for the available hardware, graph optimizations, operator fusion, and reduced precision where response quality can withstand it.
The third area is post-processing and waiting for resource release. Decoding, candidate selection, and sorting of results compete for the same thread as the run, and with parallel requests a queue is added.
Response time under peak load grows nonlinearly, so what matters is not the average latency on a single request but behavior under load — that is where it becomes clear what to optimize first.
What gets faster: latency, throughput, load
Inference acceleration breaks down into three independent quantities, and changes rarely improve all of them at once.
A node on ONNX Runtime with a tuned execution provider and batching handles many times more requests per second than the same node with default settings: compute kernels stop idling between calls, and input data is grouped so that memory is read continuously.
We measure the throughput of a single node before and after changes — this is the first number that shows the effect.
Average latency is deceptive: it masks the tail that determines user experience.
That is why we look at the median and the 95–99th percentile separately — the tail shrinks more noticeably than the median, because what goes away is not average work but rare expensive branches: graph rebuilding, extra tensor copies, waiting in a queue.
Hardware utilization shows exactly where time is lost. Low compute utilization with a full queue means work is stuck on data preparation; the opposite picture is a memory bandwidth bottleneck. When these profiles are balanced, a single node handles the same traffic without adding hardware.
A separate result is an eliminated class of failures. Under peak load, requests accumulated in a queue, timeouts triggered, clients retried calls and intensified the load.
Throughput headroom and predictable latency remove this cascade entirely, and warm-up after startup stops consuming the first requests: warm-up runs and a reusable cache of compiled graphs make the node ready for load immediately after startup.
Acceleration modes: comparison of approaches and limitations
Acceleration modes do not replace each other — they are layered, and each subsequent one requires headroom in memory or accuracy. ONNX Runtime enables them separately, but the choice starts with what exactly is the bottleneck: computation time, model size, or waiting in a queue.
| Mode | When it fits | Limitations |
|---|---|---|
| Computation graph optimization | Ready model, weights cannot be changed, gains needed without retraining | Helps only on operator groups that can be fused; on a sparse graph the effect is close to zero |
| Weight quantization | Not enough memory or need to run on CPU without an accelerator | Accuracy drop on sensitive layers — testing on your own set is mandatory; some operations remain in full precision |
| Request batching | Many parallel calls and memory headroom | Latency of an individual response grows; it makes no sense for single requests |
| Cache of repeated requests | Template calls: reference answers, labeling, classification | Useless for unique requests; a reset policy and storage space are needed |
| CPU only, without an accelerator | Accelerator unavailable or load is infrequent | Latency is an order of magnitude higher, and the ceiling of concurrent requests is lower |
In practice, modes are combined: compressed weights together with graph optimization give the greatest gain, the cache handles repeated calls before computation, and batching is enabled where response latency is not critical.
We run each combination on your set and compare answers with the full-precision reference — otherwise the acceleration will be paid for with silent model errors.
Work stages: from bottleneck audit to handover of the result
We start with measurements, not with editing configuration: without a profile of the current inference, any optimization turns into guesswork.
We measure how much time goes to pre-processing, the model pass itself, and post-processing, and only after that do we decide what to change. Below is the order of work that every project follows.
- Performance audit and measurements. We capture a profile on a real load distribution, not a synthetic one. We break down time by stages, look at accelerator utilization, memory leaks, and request queues — the bottleneck becomes visible before work begins.
- Deployment scheme. We determine where the model runs: on CPU, on accelerator, or hybrid; how many copies we launch; how we group requests into batches; and what we keep in cache. The scheme is defined before tuning; otherwise optimization hits an architectural limit.
- Inference tuning. We convert the model to the ONNX Runtime intermediate representation, enable quantization, and select the execution provider and operator fusion. We change parameters one at a time, measuring after each step.
- Reference request set. We collect requests from real operation, including long and edge-case ones. On them we calculate latency and throughput before and after — the numbers must be reproducible on your side.
- Response quality control. We compare the output of the optimized and original models on the reference set, record the match rate, and analyze deviations. If quality falls below the agreed threshold, we roll back the configuration and look for another mode.
- Handover and team training. We hand over configurations, measurement reproduction scripts, the model update procedure, and an analysis of typical failures. We conduct training: your engineers must be able to repeat the measurement and deploy the build without us.
The order is essential: measurements before tuning, quality checks before handover. Otherwise, faster response turns into regressions that surface in production — and rollback has to be done on a live service.
Handover package: configurations, diagrams, and procedures
The project does not end with a working service — your team retains a set of artifacts that allow the system to be updated, scaled, and restored without us. Below is exactly what we hand over and why each document is needed in practice.
- Environment configurations — startup parameter files, memory and CPU thread limits, and request batching settings. With comments: which parameter to change as load grows, and which must not be touched without retraining the model.
- Deployment diagram — service topology, where the model lives, where the cache is, which nodes scale horizontally, and which require reserve capacity.
- Access policies and roles — separation of rights: who deploys a new model version, who reads logs and metrics, who changes configuration. With a role matrix and description of service accounts.
- Operations documentation — startup and shutdown procedure, service health checks, and analysis of typical failures with symptoms and on-call actions.
- Version update and rollback procedure — the order for deploying a new model build, the time to return to the previous one, and the conditions under which rollback is triggered automatically.
- Set of metrics and thresholds — which indicators we measure, at what values alerts trigger, and how to distinguish response quality degradation from a network failure.
- Environment reproduction guide — build manifests and dependencies, so the test bench can be deployed from scratch on another machine.
We write each document for a specific performer: an on-call engineer, a developer, or a manager. Such a package removes dependence on the contractor — changes are made by your team, and we remain available for complex cases.
Case study: how we cut model response time by half
The task came from an internal support service: employees asked a question against the knowledge base and waited for an answer in a chat window.
At peak, the working node went to full utilization, the request queue grew, and some requests dropped due to wait time — people returned to manual document search.
We started the analysis with measurements: we captured how much time goes to data preparation, the model inference itself, and response assembly.
The main time was consumed not by the model, but by constant rebuilding of the computation graph for each call and by processing requests one at a time.
We moved the graph into ONNX Runtime, fixed the input shape, enabled operator fusion and a cache of intermediate states, and converted weights to integer representation.
Measurements before and after
- model response time — from 3.2 to 1.3 seconds on the same hardware;
- peak node utilization — from 94 to 58 percent;
- number of nodes for the same request flow — half as many;
- share of answers matching the reference labels — 93 percent before and after.
The service handles peak flow without queueing or failures, and the node headroom allows load to grow without urgently purchasing servers.
The client received a reproducible measurement loop: each subsequent run is compared against a fixed baseline, so the effect of any optimization is visible immediately.
What to do if the model is already deployed and responds slowly
A deployed model rarely slows down because of architecture. More often, time goes to pre-processing, queue waiting, response serialization, and unnecessary recomputation, while inference itself takes a smaller share.
Therefore, the first step is to break down latency by stages: measurements at the input, at the model output, and at delivery to the client show exactly where time is lost, and only after that is it decided what to touch.
If the contract cannot be changed, work is done at the environment level, and the application code remains the same.
The model is converted to an intermediate graph representation: ONNX Runtime inference optimization provides a choice of execution provider, and heavy operations go to the accelerator, while the rest remains on CPU.
The number of worker threads, batch size, and graph optimization level are set in configuration — these are settings, not rewriting logic.
Next comes the volume of computation and repeated work. Weight quantization reduces model size and speeds up the forward pass without retraining; reuse of computed context eliminates recomputation at each generation step; batching smooths load under peaks.
Each step is reversible, and answers before and after are compared — the divergence shows where quality dropped.
This is rolled out gradually: the optimized variant runs alongside the current one, receives a copy of the traffic, and metrics are captured on live requests — latency, failure rate, answer match.
Switching is done by shifting traffic, and rollback is done by returning the previous artifact, so the risk is limited to one release. Inputs, outputs, and the response schema do not change, and external services do not notice the runtime replacement.
Addressing common technical doubts from customers
Inference optimization comes down not to tool choice but to verifiability of the result: it is important to know exactly where quality is lost, whether the measurement will repeat on another bench, and what will happen to an already working service. ONNX Runtime here is the executor, and decisions are made based on metrics.
Does quantization break model accuracy?
Converting weights to 8-bit representation does change the output, so the loss is measured, not assumed. We run the reference request set before and after optimization and compare response quality with the divergence rate.
If degradation exceeds the agreed threshold, we leave sensitive layers in full precision and accelerate only the heavy ones — speed increases, and quality remains within the target range.
How is the result verified, not just speed?
We assemble a regression set from your real requests with reference answers. Each configuration variant is run on it, both quality and latency are recorded, and the choice is made along the quality–speed curve, not from one lucky measurement.
Are measurements reproducible on another machine?
A measurement makes sense only if it repeats. We fix the model version, generation parameters, set composition, and warm-up mode, run a series of times, and look at the variance.
The report retains the bench description and the launch command — you recheck the numbers on your side.
What about hardware requirements and rollback?
CPU acceleration does not require separate accelerators, while the variant with them gives a gain only under sufficient load. First we capture a profile of the current load and select a configuration for it.
We make the transition in stages: the new configuration is enabled alongside the old one and rolled back by switching without rebuilding the service.
Let's discuss the task: we will prepare a plan and estimate
For the work plan to be realistic rather than generic, we need a short context: where the model currently runs, what response to a request is considered acceptable, how many calls arrive at peak, and whether there are requirements to keep data within the perimeter.
This is enough to propose a specific inference optimization scheme — on CPU or accelerator — and provide timelines by stage.
- What we need from you: the use case — generation, classification, or knowledge base search — plus current latency and throughput measurements.
- Hardware constraints: which CPUs and accelerators are available, whether machine configuration can be changed, and whether there are memory limits.
- Perimeter requirements: where data may be processed, whether there are isolated segments, and which access rules apply.
- What we return: a stage-by-stage work plan with timeline estimates for each — from taking measurements and preparing the model to rollout into the production environment.
- Interaction procedure: one point of contact on our side, a status once a week by email or a short call, and access to the test environment for verifying the result.
- Acceptance criteria: latency and throughput values fixed before the start, by which we consider the result achieved.
- Rollback plan: how we return to the original configuration if after rollout the metrics turn out worse.
Describe the task in free form — we will return with a work plan and timeline estimate. If the inputs are not enough for precision, we will ask targeted clarifying questions and immediately indicate which of them affect the duration.
