Why transactions fail in production and how Tenderly Debugger helps
A transaction reverts on mainnet, and the node returns only the status and a generic error code — without the reason.
Gas is spent, the state has not changed, and guessing begins: wrong argument, stale oracle, gas limit on a nested call, or a third-party protocol that returned a rejection deep in the stack.
The cause almost always lies not in the top-level code, but in the call chain beneath it — through proxies, libraries, and external contracts.
Conventional tools are almost useless here: the receipt shows only the fact of failure, simulation without the exact block state gives a different result, and a local run does not reproduce balances, storage, and sender address at the time of failure.
Debugging smart contracts turns into trial and error: comment out a section, deploy to a testnet, repeat.
Tenderly Debugger removes this information vacuum. Call tracing shows all frames — from the entry point to the last nested call — along with arguments, storage changes, gas usage, and the exact revert location.
It shows who exactly returned the error: your check, a condition inside a library, or an external protocol that ran out of gas under the remainder-forwarding rule.
Then the failure is reproduced on a mainnet fork: you can substitute balance, storage, or sender and run the scenario again without touching production.
Instead of hours of trial and error — a specific frame and line where the logic breaks, even before the next version is deployed.
What transaction debugging provides: investigation speed and class of reverts
Debugging is not a convenience for the developer, but a way to remove uncertainty from investigating a contract incident. While the cause is sought by trying hypotheses against logs, every failed transaction remains a mystery, and the downtime window grows.
Tenderly provides step-by-step call tracing with the state at the time of failure — that is why the effect is measurable, rather than “it became more convenient”.
- Error localization time — instead of hours spent reconciling logs and events: tracing shows the specific nested call where the revert occurred, and the arguments passed into it. The cause is found in minutes.
- Share of reproducible reverts — a state snapshot at the required block allows replaying the incident in an isolated environment as many times as needed. Some defects are no longer closed “by description” — only by a confirmed scenario.
- Eliminated class of errors — rounding, incorrect call order, gas exhaustion stop repeating: each investigated case turns into a regression check.
- Downtime cost — when the cause is known and reproducible, the service unavailability window shrinks: the fix and rollback are prepared on a verified scenario, rather than in a hurry.
- Investigation transparency — every incident leaves a trail: input data, trace, cause, fix. Repeated “why did this fail again” disappears.
Debugging methods: simulation, network fork, and step-by-step analysis
The debugging method is chosen based on the moment when the defect is found: before sending a transaction, when reproducing on a copy of the network state, or after the call has already executed. Below are four working approaches and the boundaries of applicability for each.
Transaction simulation and a network fork cover most tasks before deployment: the first runs the call against the current state without writing to the chain, the second provides a copy where balances and block time can be changed.
Once the call has executed, node log analysis and step-by-step tracing remain — they show exactly where the failure occurred.
| Approach | When it fits | Limitations |
|---|---|---|
| Transaction simulation | Testing a hypothesis before sending: contract edit, gas calculation, replaying a production call | Only the selected network state is visible; private nodes are connected separately |
| Network fork | Scenarios with multiple contracts, balance and block time substitution | Snapshot becomes stale — the fork is rebuilt on fresh blocks |
| Local node | Full control over state, tests of migrations and private data | Production state is reproduced manually, discrepancies are possible |
| Step-by-step tracing | Failure point, internal calls, gas usage step by step | Large output volume, filtering by addresses and calls is needed |
| Node log analysis | Transaction has already executed, it cannot be changed | Only what the contract itself wrote to events is visible |
The approaches do not compete: simulation gives a quick answer to a hypothesis, a fork gives the whole scenario, tracing gives the cause of a specific failure.
In Tenderly this is combined: the same call can be run in simulation and then analyzed step by step without transferring data between tools.
Setting up Tenderly Debugger: work stages from audit to handover
Setting up Tenderly Debugger is not a one-time tool installation, but a sequence of work after which the team reproduces the failure of any transaction without manual trial and error.
We start with an audit: we look at how the cause of a failure is currently sought, how much time reproduction takes, and what data is lost in the process.
- Audit of the current debugging process. We analyze how the team handles failed transactions: export from the node, rerun, reconciliation with logs. We record exactly where storage state and internal calls are lost.
- Environment scheme. We mark out networks — mainnet, testnet, local copies — and organize them by projects so that experiments do not mix with production investigation. Each environment gets its own set of access rights.
- Network connection. We configure access to archive nodes: without state at the required block height, tracing cannot be built. We connect non-standard chains manually and verify that the debugger shows the same result as the node.
- Trace access. We separate roles and keys: a developer sees only their own projects, release checks use a separate key. We store secrets in a secure vault, not in the repository.
- Validation on production data. We run real failed transactions, compare the trace with node logs, and catch discrepancies in gas usage and internal transfers. Done when the failure reproduces on the first run.
- Team handover and training. We hand over environment descriptions, investigation templates, and breakdowns of typical errors on live cases, so that debugging can be carried out in-house going forward.
Artifacts and documentation: what remains in your repository
We hand over not a report on work performed, but a working set that remains in your repository. With it, the on-call shift investigates a failed transaction without the author of the change and without chats.
- Environment configurations — connection files for local, test, and production networks and check sets for each environment; changing environment does not require code edits.
- Environment and network scheme — which environment is tied to which network, where the state clone is raised, and where work happens on real data.
- Contract map — who calls whom, where proxies and libraries are, which addresses change when the implementation is upgraded.
- Debugging procedure — the order of investigation in Tenderly Debugger: simulation at the required block, tracing, analysis of the revert cause and state before and after the call.
- Access policies — who sees production data, who can run a replay, who grants and revokes rights; the issuance log is maintained and stored.
- On-call instructions — a checklist for the first minutes of an incident: what to capture, what to attach to the request, and when to involve the developer.
- Handover documentation — description of environments and entry points, known limitations, analysis of two or three representative failures, and a recording of the meeting with the team.
A month later, a new engineer conducts the investigation using these materials on their own — without contacting the person who set up debugging.
Case: failure investigation dropped from hours to minutes
A protocol on an EVM-compatible network was receiving transaction reverts from users, while node logs only retained a generic execution reverted without a reason.
Support collected hashes manually, the infrastructure team brought up an archive node and ran the investigation in a separate sandbox — each incident took 4 to 6 hours, and almost half of them were closed by resubmitting without understanding what exactly failed.
What we changed
We built a full call tree with internal transitions, logs, and state changes, tied snapshots to a specific block and configured transaction replay on a network fork along with balance history.
In Tenderly, each revert opens via a link from the support ticket: the engineer immediately sees at which call, with which arguments, and at what gas remaining the logic reverted.
We added threshold alerts by revert reason — a spike of similar triggers arrives before the flow of support requests.
Metrics before and after
| Indicator | Before | After |
|---|---|---|
| Failure cause localization | 4–6 hours | 10–20 minutes |
| Share of reproducible incidents | about 30% | more than 90% |
| Investigation without revert reason | every second case | isolated cases |
A whole class of reverts was closed separately: mass failures at the swap confirmation stage due to a discrepancy between the expected and actual amount.
The cause was found from the trace in a single pass and removed in the contract with an input check, rather than a patch in the interface.
The share of user transaction reverts on the network fell by about three times — tracing became a standard step before every release, and releases no longer go out with untested execution paths.
How is simulation different from a network fork during debugging?
Simulation is a single run of a call against the state fixed at a specific block. It catches deterministic reverts: insufficient allowance, an access limiter triggered, execution deadline expired, price moved beyond the allowed deviation, wrong argument order.
The result is reproducible — you can see at which internal call the error return occurred and which storage slots changed before and after.
Network fork is a copy of the state at the selected block, in which transactions run sequentially and accumulate effect.
It is needed when the failure depends on previous transactions, the oracle state at the time of the call, or on someone having managed to change the pool earlier. Non-persistent failures usually reproduce only here: a single run either does not show them or shows them unreliably.
The incident dictates the choice. One error return and a clear call stack — simulation is enough, if necessary with state overrides: increase balance, rewrite a slot, substitute the sender.
The failure repeats or manifests only after a series of actions — we bring up a fork at the exact block from the logs and restore the call order.
The practice is as follows: first a fork at the incident block, record the chain, then minimize it to a single simulation — this way the cause stops being “floating”.
Tenderly provides both modes in one interface, so switching between them does not require rebuilding the environment. An important caveat: both simulation and fork need archive state, otherwise the required block simply will not come up.
Method limitations and frequent tracing questions
Tracing shows execution step by step, but source code is not always returned: the optimizer collapses calls, and without a build map the trace reads like a stream of instructions.
Simulation on a fork reproduces the state of contracts at the snapshot moment and does not catch dependencies on transaction order in the queue. These are the method's boundaries, and below are the questions that most often come up before work starts.
The contract is already deployed — is redeployment needed?
No. Debugging in Tenderly works by address and transaction hash; the contract does not need to be rebuilt. If the sources are not published, we load the build map — trace steps again read as function calls and variable changes.
Are private keys and access to your account needed?
No. Tracing runs in read-only mode: access to the network node is enough, and for historical transactions — to the archive node. Signing keys do not leave your environment; we do not send transactions.
What to do if the network is deployed in a closed environment?
We bring up an isolated debugger instance inside your perimeter, without calls to external services. It synchronizes with your node; only the reports you allow to be exported go outside.
How is debugging data stored?
We define the retention policy at the start: raw traces can be not stored, leaving summaries of failures. Personal data and secret values at the input are masked; retention period and access levels are fixed in the procedure.
Let's discuss the task: where to start and what is needed as input
The more accurate the input data, the shorter the path to working debugging. To come back with a work estimate and implementation plan, rather than clarifying questions, we need the minimum for your environment.
- Contract addresses and the networks where they are deployed: mainnet and testnets, marked which are critical for investigation.
- Five–ten hashes of failed or suspicious transactions — they show what data is missing in the current trace.
- Contract sources or ABI, including proxies and the logic behind them: without this, internal calls remain unnamed.
- Description of the access environment: how node access is granted, who owns the keys, whether there are restrictions on outbound connections.
- Current debugging scheme: what you use, where you look at logs, what you have already tried. If investigations are already conducted in Tenderly Debugger — attach links to saved sessions.
- List of networks and expected flow: how many transactions per day pass through the service.
- Person responsible for infrastructure and the window during which configuration changes are allowed.
Next, we conduct a debugging audit: reproduce errors, see at which step the call context is lost, check behavior on a copy of the network state. After that, we come back with a work estimate and implementation plan — by stages, with checkpoints at each.