dApp Backend Development: Where Network Synchronization Breaks
A dApp backend sits between the interface and the distributed network, and it is the most fragile part of the system. A node response arrives with a delay or does not arrive at all, while the application already shows the user success.
Until this is addressed, logic on top of the chain relies on a state that does not yet exist in the network.
A separate class of failures is stuck transactions. The signature has gone to the network, the fee turned out to be below the current market rate, and the operation hangs for hours, blocking the queue of subsequent ones.
Without status tracking and resubmission with revised parameters, the user does not understand whether anything was debited, and support cannot give any definite answer.
Duplicate operations appear where the notification handler restarts mid-processing or the network returns the same event again. The same transaction is counted twice, balances and limits diverge from the actual chain state.
Idempotency by transaction hash and a separate storage of processed events eliminate this class of errors entirely.
A mismatch between the interface and the chain is compounded by a reorg: the block on which the calculation relied drops out of the main branch. This is treated not with cosmetic fixes but with a single read point with a fixed confirmation depth and explicit operation states.
We build such services on web3.py — dApp backend development here starts with a failure model, not with screens.
Below we examine how network synchronization, node unavailability, and duplicate operations are addressed.
What a Dedicated Backend Provides: Response Speed and Resilience
A dedicated backend for a decentralized application is needed so that the interface responds to user actions without waiting for final confirmation in the network.
Requests to read state and prepare operations are served from an in-house index and local cache, so response time is measured in fractions of a second rather than seconds of waiting. The network remains the source of truth but stops being the point that every click runs into.
Service resilience rests on idempotency: each operation has its own key, so repeated submissions during a connection loss do not create duplicates — the action either succeeds once or returns an already known result.
This closes the class of failures that is most costly in operation: double debits, lost state between requests, and "hanging" operations without a clear status.
Peak processing is moved to a rate-limited queue with priorities. When activity spikes, the service does not crash or lose tasks: it spreads processing over time, and the interface shows the real status of the operation instead of a false "completed".
Unconfirmed actions are retried with increasing delay and an upper limit — this dampens the request storm during bulk operations.
The practical result: the share of scenarios that pass without manual intervention is above ninety percent. The operator handles atypical cases, not every network failure.
Synchronous Calls, Task Queues, and Event Streaming: Comparison
A direct synchronous call to a node is the simplest path: the response arrives in the same request, and no separate infrastructure is needed.
The limitation is that the response depends entirely on an external service, and a connection drop immediately after sending a transaction leaves uncertainty — whether to consider the operation completed or retry it.
Background processing through a task queue separates request acceptance from network work: the client receives an operation identifier and polls for the state. Event streaming gives the opposite picture — the service reads the block log and initiates the reaction itself.
In practice, a hybrid scheme more often wins: writes go through the queue, while state is assembled from an event stream.
| Scheme | When It Fits | Limitations |
|---|---|---|
| Synchronous calls | Rare operations, state reads, manual administrator actions | Response depends on an external service, no retries on connection loss, on peak requests grow in handlers |
| Task queue | Transactions that cannot be executed at request time: payouts, mint, batch operations | Delay between request and result, an idempotency key is required, otherwise a retry will create a second operation |
| Event streaming | Continuous accounting: balances, history, notifications, indexing | Complex rollback during chain reorganization, need to store the read position and catch up on missed items |
| Hybrid scheme | Production service where reads and writes follow different rules | More code and failure points, explicit separation of responsibilities between the request loop and workers is required |
The difference shows up during failures, not in calm mode. That is why the scheme is chosen based on load, response time requirements, and how critical operation retry is — while the queue, stream, and direct connection in one service are routed along different paths.
Work Stages: Audit, Design, Setup, Verification, Handover
The work is divided into stages so that each ends with a verifiable artifact: an integration map, a design, a configuration, or a load report. The project state is visible at any step — discrepancies with expectations are identified before changes go into the main branch.
-
Integration audit. We capture the current state: which nodes read the network, how transactions are formed and signed, where keys are stored, how block reorganizations and repeated event deliveries are handled. The output is a list of bottlenecks with risk assessment and a list of what we will reuse.
-
Service design. We break the backend into layers: request acceptance, event processing, task queue, state storage. We determine where idempotency is required, how reads and writes are separated, and which keys deduplication is built on.
-
Queue setup. We distribute tasks by priority and separate queues: transaction confirmation, indexing, delayed retries. We set concurrency limits and a retry policy so that a stuck task does not block the others, and "poison" messages go to a separate review.
-
API and configuration assembly. We describe contracts, input validation, idempotency keys, and request rate limits. Secrets and environment parameters are moved out of the code.
-
Load verification. We replay real event recording and a synthetic stream, measure the delay from block appearance to service response, and check behavior during node failure and queue overflow.
-
Project handover. We deliver the repository, migrations, deployment scheme, metrics with alerts, and incident response procedures. We conduct a walkthrough with the client's team so that further development can proceed independently.
What You Keep: Designs, Configurations, and Procedures
Handing over a project without artifacts quickly turns it into a black box: the team has left, and there is no one left to figure out why the worker is not catching up with the network or how to rotate the signing key.
That is why handover is a separate work stage, not an archive at the end. Here is what remains with you in the repository and internal knowledge base.
- Environment configurations — variable files for the local stand, test, and production environments; secrets are separated, and each file has an example of how to fill it in. A copy of the service can be brought up without correspondence with the authors.
- Architecture designs — how nodes, queues, and processing workers are connected; behavior when a component fails and where the retry goes are noted separately.
- Access policies — roles and permissions, separation of signing keys and read-only keys, rules for secret storage and their rotation procedure.
- Update procedure — how a new version is released: build, test run, deployment, rollback window, and who decides to stop a release.
- API documentation — methods, request and response schemas, error codes, limits, and call examples for your client application or a third-party service.
- Operations instructions — starting and restarting workers, reading logs, actions when state is out of sync with the network, and typical failures with ready-made solutions.
- Configured monitoring — alerts about processing delays, write errors, and insufficient funds in the service account, with trigger thresholds and recipients.
Before handover, we bring up the stand from scratch using the same documents on a clean machine: if a step cannot be reproduced, the instruction is returned for revision.
Case Study: How We Eliminated Downtime Under Peak Load
The client is a blockchain request service: the user signs an action, and the backend sends a transaction and monitors its inclusion in a block.
Request acceptance, the task queue, and sending were handled by a single Python process with web3.py: under a stream of 3–5 thousand concurrent sessions, the node responded with delay, the queue swelled, and some signed transactions never reached the block.
What Was There
The share of lost operations was 4–6% at peak hour: submissions with expired lifetimes and overwritten nonces settled in the queue and were simply discarded.
Recovery after a node crash took 40–60 minutes of manual work: bring up the service, reconcile wallet states, and finish stuck submissions. During that time the interface returned timeouts, and users pressed "send" again, creating duplicates.
What We Changed and What Improved
We moved the queue into a separate layer with a length limit and priorities, made submission idempotent by request key, and added our own nonce accounting with signature recalculation on retry.
Nodes were connected with auto-switching, and the state of each transaction is stored until final confirmation — a service restart no longer loses anything.
The share of lost operations dropped to 0.1%, and these are only those rejected by the network with an explicit reason.
Recovery after a node failure takes 90 seconds and requires no engineer involvement: traffic goes to the backup, and stuck submissions are replayed from storage.
At peak hour, the interface sustains the design load, duplicates are filtered out at the input, and the queue and confirmations are visible on graphs.
How Your Own Network Node Differs from a Third-Party Provider
A third-party provider is a subscription gateway to the network: calls go through a shared pool, not through your infrastructure. Startup takes hours, but service behavior depends on someone else's scheduler — at peak, a 429 rejection arrives and retries instead of a fresh block.
Request limits are counted by method and second, so heavy queries over a large block range are cut first.
Your own network node removes request limits entirely: throughput is limited only by hardware, and you decide how many parallel event subscriptions to maintain.
This is critical when an indexing service continuously reads logs while background tasks catch up on lagging blocks in parallel.
Then infrastructure load begins. A full node requires a fast disk and stable synchronization, while an archive node stores the states of all blocks and grows linearly.
On the other hand, recalculating history after a failure, reconciling balances, and exporting for an arbitrary period become routine rather than a separate project.
In practice, most services operate in hybrid mode: their own node covers the hot path — sending transactions and reading fresh blocks — while Infura remains a backup during client updates or lag.
In code, this turns into a call queue, batching requests, a header cache, and background state checks that switch the source when it falls behind.
Technical Questions: Nodes, Limits, Resubmissions
An asynchronous backend for a decentralized application does not live in a vacuum: public nodes respond with delay, return an error when the request limit is exceeded, and sometimes lose connection for tens of seconds.
We place a pool of providers with automatic failover in front of the service, and each submission to the network gets its own identifier — restarting the handler does not create a second transaction.
Request limits become a predictable configuration parameter rather than a source of failures at peak hour.
Incomplete operations are not lost either. The journal stores the state of each task, and after a restart the service reloads what remains in the "sent, no confirmation" status, verifying the result by hash. The user sees the current operation status, not "stuck".
What happens if the provider node starts returning errors?
The service switches to a backup node from the pool, and the task returns to the queue with a longer pause between attempts. If all are unavailable, operations accumulate in the journal and complete after connectivity is restored, without manual intervention.
How are signing keys stored?
Private keys do not enter either code or logs: they are kept in isolated storage, and the service requests a signature through a narrow interface with a limited set of operations. Access is role-based, and every request is written to an audit log.
We configure monitoring before launch: queue depth, share of failed submissions, confirmation delay, and degradation notifications. Dependency updates follow a separate procedure — first a testnet run, then deployment, so that a library version change does not break contract interaction.Let's Discuss the Task: We Will Return with an Estimate and Work Plan
For the estimate to rely on facts rather than assumptions, a short set of inputs is needed — usually one conversation and access to what already exists is enough: the repository, contract descriptions, and environment design.
We work with the Python stack for dApp backend, so we will immediately say which components will require rework and which can be reused without rewriting.
What we need from you to start:
- Which contracts are already deployed, on which networks, and which operations the backend performs on-chain.
- Who signs transactions and how key storage is arranged — the access and rollback model depends on this.
- Expected load: how many operations per day, where the peaks are, and how critical response delay is.
- Environments: where we deploy, whether there is a test network, and how releases are deployed.
- Data handling: whether event indexing, operation history, exports, and reporting are needed.
- What has already been written: repository, description of bottlenecks, known bugs, and open tasks.
- Who on your side will approve the architecture and stage priorities so as not to slow down the work.
Based on the discussion of the task, we will return with an estimate, an implementation stage plan, and team composition: who leads the backend, who is responsible for infrastructure, and who handles testing and acceptance.
Describe what already works, what should start working, and by what deadlines — and we will return with a plan.