Why on-chain AI inference breaks production
Putting model inference on the blockchain runs up against not the quality of the model, but the nature of consensus.
Every inference recorded in the network state goes through validation by all nodes — and while the block is being finalized, confirmation latency is measured in seconds and tens of seconds, not the milliseconds that are customary in an ordinary service.
For a chatbot or real-time scoring, this is already unacceptable: the user is waiting for a response, while the transaction is still hanging in the mempool.
Gas costs turn this into an economic problem. Heavy model inference means hundreds of thousands of operations, and the price of on-chain computation grows linearly with their number.
Under peak load, the fee for a single call can exceed all the revenue from it, and a network under load begins to push out expensive transactions. Worse still, gas costs are unpredictable: what yesterday fit within a reasonable budget today breaks the deal.
Model drift finishes off the picture. Weights fixed in the contract are not updated without a state migration — which means a model trained on past data keeps producing answers that no longer match reality.
Retraining requires a new version of the contract and migrating the history, and all that time the system runs on outdated logic.
A separate pain point is incompatibility with oracles. External data arrives asynchronously, with its own latency and risk of manipulation, while the contract waits for it synchronously.
The desynchronization between the chain state and the input data breaks determinism: two nodes can get different answers to the same request.
Let's break down how this is solved — with a hybrid architecture where computation is moved outside the chain, and only the verifiable result remains on-chain.
What moving inference to the blockchain provides
Verifiability of outputs instead of trusting the operator
Today, inference almost always runs through a closed service: the model, weights, and post-processing logic belong to the provider, and the customer only receives the answer.
There is no way to verify it — neither to reproduce it nor to prove to a third party that the model was not swapped and the input data was not distorted.
Moving inference to the blockchain eliminates this entire class of failures: every output is recorded as a transaction, and the result can be recomputed from the same input data to obtain an identical answer.
The dispute over "who is right" turns into a check that any network participant can perform.
Recovery after a node failure
In a centralized scheme, the failure of a compute node means downtime until a manual restart and the loss of the accumulated queue state.
In decentralized inference, the task is already decomposed across independent executors: if one fails, its share is redistributed among the others, and intermediate results are stored in a shared log.
Recovery time is reduced from hours to minutes, and this does not require an on-duty engineer — the network keeps computing while the node comes back online.
Economics and scale
Decentralized inference makes it possible to scale capacity without rebuilding the architecture: new executors connect to the same network and take on the load. The customer pays for the computations actually performed, not for idle reserves held for peak demand.
The transparent log has another consequence — inference costs stop being a black box: you can see how many tasks were solved and what resources went into it.
What this changes for the product
Verifiability of outputs opens up scenarios that previously required a trusted intermediary: auditing model decisions, settlements between multiple organizations, regulatory reports.
Eliminating the class of failures involving trust in the operator removes the main blocker for such deployments — now the result can be presented, not merely asserted.
Inference Architecture Options: A Comparison
The choice of architecture determines not only the model's response latency, but also what guarantees the consumer of the result receives.
Fully on-chain inference means that every computation step is reproduced and verified by the network — this is maximum trust, but throughput runs into block limits, and cost grows linearly with the number of calls.
The hybrid approach moves heavy computation outside the network, while the result is recorded via an oracle: the network stores only the input, the output, and the executor's signature.
The third option is off-chain computation with proof verification. The model runs on the operator's side, but publishes a cryptographic proof of correctness that any participant can verify without re-running it. This relieves the network while preserving the ability to challenge an incorrect result.
| Option | When it fits | Limitations |
|---|---|---|
| Fully on-chain | Small models, critical audit of every step, low request frequency | Network throughput, cost growth with the number of calls, response latency |
| Hybrid with oracle | Production load, need for a balance of speed and trust | Trust in the oracle and executor, a challenge mechanism is required |
| Off-chain with computation proof | Large models, infrequent but expensive computations | Complexity of proof generation, verification time, maturity of tooling |
In practice, pure options are rare: more often they are combined — a light model and aggregation of results on-chain, the heavy part with a proof. We select a scheme for the load profile and verifiability requirements, not the other way around.
How We Implement an Inference System: Stages
Implementing inference in a blockchain environment starts not with code, but with an inventory: which models are already running, how many requests per second they handle, which responses are latency-critical, and which can tolerate asynchronous processing.
At this stage, we identify the points where computation must be deterministic, otherwise validators will diverge in their results and consensus will not be reached.
A model audit shows what can be moved into a verifiable circuit and what is cheaper to leave in an external service with cryptographic confirmation.
Designing the Scheme and Configuring Nodes
The scheme determines how a request enters the network, who performs the computation, and how the result is confirmed.
We choose between full on-chain AI inference and a hybrid option with proof of correctness — the difference lies in gas costs and in the hardware requirements for nodes.
Node configuration includes client setup, state synchronization, limits on input data size, and a retry policy for cases where results diverge.
- Audit of the current model: we measure latency, input data volume, and the share of non-deterministic operations.
- Scheme design: we distribute the computation between the contract, the oracle, and the external executor.
- Node configuration: client setup, synchronization, limits, retry policy.
- Deployment to the test network: running scenarios with diverging results and node failure.
- Test network validation: load runs, gas measurement, cross-checking responses between validators.
- Handover: configuration documentation, model update instructions, incident analysis.
What We Check Before Handover
The test network is a mandatory stage: we deliberately break one node and observe how the system restores consensus. If the divergence in responses is not resolved automatically, the scheme is sent back for revision.
The handover includes not only the deployed circuit but also the model update procedure: how to roll out a new version without stopping the network, and how to roll back if response quality drops.
Artifacts and documentation after implementation
Implementing on-chain AI inference does not end with the launch of the loop — the customer receives a set of artifacts that allow the system to be operated, extended, and handed over to another team without verbal agreements. Below is what exactly remains in hand after delivery.
- Inference node configurations: parameters of compute nodes, resource limits, request queue settings, and scaling rules for peak load — as versioned files, not manual edits on servers.
- Data flow diagram: from a client request through the smart contract to result delivery — with validation points, routing between nodes, and a description of exactly where the computation proof is recorded.
- Access policies: roles for operators, developers, and external consumers of the model, transaction signing rules, limits on contract calls, and the procedure for issuing keys.
- Model update procedure: a step-by-step order for rolling out a new version — from building the artifact and testing on a staging loop to switching traffic and rolling back to the previous version.
- Monitoring and alerts: a set of metrics on inference latency, the share of rejected requests, and divergence of results between nodes, with trigger thresholds.
- Contract documentation: a description of methods, events, and storage structure — so that an external audit or a new developer can understand it without reverse engineering.
- Recovery plan: scenarios for node unavailability or state desynchronization, indicating recovery time and responsible parties.
Each artifact is tied to a specific stage of operation, so the customer's team handles typical tasks — updating the model, connecting a new consumer, investigating an incident — by the document, not from the developer's memory. If needed, we can walk through how this set fits into your support process.
Case: threefold inference acceleration
The task was typical for teams that move model output into a distributed registry: inference was computed off-chain, and the result landed on the network as is.
It could only be verified manually — an analyst cross-checked the input, weights, and output against the logs, and each disputed request took anywhere from several hours to a day.
With a flow of hundreds of requests per day, the verification queue grew faster than the team could clear it, and the response time for a verified result reached tens of seconds.
We moved verification into on-chain AI inference: the inference result and its proof are recorded in a single contract, and the validator cross-checks them automatically against predefined rules.
Manual verification disappeared as a stage — disputed cases are no longer reviewed after the fact but filtered out at the input. We worked on an Ethereum-compatible network with batch recording of results, so as not to pay for each request separately.
Metrics before and after
| Indicator | Before | After |
|---|---|---|
| Response time for a verified result | tens of seconds | single-digit seconds |
| Manual verifications per week | hundreds | 0 |
| Share of requests with automatic verification | partial | 100% |
| Resolution of a disputed case | hours | minutes |
The threefold acceleration came not from the model output itself but from removing the manual verification queue: engineers stopped being the bottleneck.
A rollback to the previous scheme is possible — the contract is versioned, old results remain valid, so the migration does not block current processes.
How does on-chain inference differ from off-chain?
The key difference is where the model physically executes. With on-chain inference, computation happens inside the smart contract: the input data and model weights are processed within the network itself, and the result is recorded in the contract's state.
With off-chain inference, the model is computed outside the network, and only the result and a proof of its correctness make it into the contract. The first mode provides maximum transparency, the second — practical performance.
The main limitation of on-chain inference is gas cost. Every computation operation is paid for, and heavy models with millions of parameters make such computation economically unviable.
That is why, in its pure form, on-chain inference is applicable to compact models: linear regression, simple classification, threshold decision rules. For general-purpose neural networks, computation is done off-chain, and verification is brought on-chain.
Verification of the result is what links the two modes. Off-chain inference does not mean "trust the executor": the result is confirmed cryptographically.
The executor publishes the model's output together with a proof — either interactive (arbitration through disputing) or a compact cryptographic one, which the contract verifies in a fixed number of operations.
The contract does not recompute the model; it verifies the proof — this is orders of magnitude cheaper.
The gas payment falls on whoever initiates the transaction. With on-chain inference, the user pays for every computation step.
With off-chain inference, gas goes only toward publishing the result and verifying the proof — a fixed and predictable amount that does not depend on the size of the model.
Hence the practical rule: the heavier the model, the more off-chain inference with on-chain verification wins.
The choice of mode is a choice between the cost of computation and the cost of trust. If the model is compact and the result is needed within the network itself — we compute on-chain.
If the model is heavy — we compute off-chain, and bring verification into the contract, which guarantees that the result has not been substituted.
Common technical concerns
When a model or weights are updated, an already deployed node does not need to be taken out of service entirely.
The new version is brought up alongside the old one, passes validation against a control set of requests, and only after the results match is it moved into the main loop.
If the discrepancy exceeds the threshold, traffic stays on the previous version — rollback takes minutes, not hours.
What should be done with an already deployed node when the model changes?
The node is updated using a blue-green scheme: a second instance with the new version is brought up alongside it, a small share of requests is routed to it, and the metrics are compared against the reference.
When the discrepancy is within tolerance, traffic is switched over entirely, and the old instance remains a hot standby until stability is confirmed. This way, the update does not require stopping the service and does not break active sessions.
Can the model be updated without downtime?
Yes, provided that the inference state and the cache of intermediate results are moved outside the model process. Then the new instance is warmed up in advance, accepts traffic after validation, and the old one is shut down smoothly — as current requests complete.
Downtime occurs only where the model keeps state inside itself and cannot expose it externally; this is designed into the architecture from the start, not fixed during an update.
What are the throughput limits of a deployed node?
Throughput is limited not by the network itself, but by the cost and latency of result validation: every output must be confirmed, and the confirmation must fit within the block window.
In practice, a node sustains a steady stream of requests with batch processing and queues, while peak bursts are smoothed by a buffer.
If the load grows by a multiple, it is not a single node that is scaled but a pool with load balancing — then the ceiling shifts without rewriting the logic.
All three scenarios — updating, avoiding downtime, and the load ceiling — are designed together, because they influence each other. Describe your task: we will advise on how to break down the model update and node scaling so that the service does not stop.
Let's discuss your inference task
To come back with an estimate and an implementation plan rather than generalities, we need to understand your current architecture: where the model is computed, how input data is formed, who confirms the result and how, what the requirements are for latency and gas cost.
Describe the task in free form — then we will ask targeted questions and propose a working on-chain AI inference loop for your scenario.
What you will get as a result:
- A breakdown of the current architecture: where inference is computed now, what the bottlenecks and trust points are.
- A target loop design: on-chain or hybrid, where weights and data are stored, how computation correctness is verified.
- A phased implementation plan with priorities and dependencies between them.
- An estimate of work by timeline and team composition for your volume.
- A list of risks: gas cost, latency, access, rollback on failure.
- Migration options from the current solution without stopping the product.
- Acceptance criteria by which you can verify the result.
Describe the task — we will come back with an estimate and a work plan.