Why a trading bot crashes and stays down
A trading bot rarely stops gracefully — it crashes according to one of four scenarios, and each looks different.
A connection failure with the exchange interrupts the quote stream, but the strategy loop keeps spinning on the last known price: calculations run on stale data, and orders are sent at a price that no longer exists on the market.
A process hang is more dangerous than an obvious crash. The process is listed as alive and responds to a check, but the trading loop has stalled on a blocking call — the trading bot's stop is recorded neither by monitoring nor by logs.
An exception in the strategy or a calculation that has consumed all memory brings the bot down completely: the connection is closed, while the state on the exchange remains unchanged.
While the bot is silent, the open position lives its own life. Stop orders that were held virtually inside the strategy are not moved by anyone, exposure grows along with market movement, and the discrepancy between local state and actual orders accumulates.
After a manual restart, the bot inherits a portfolio it did not create and treats it as a clean slate.
At night and on weekends, such a failure costs hours of downtime: the on-call engineer learns about it from drawdown rather than from an alert, and by the time of intervention the market has moved far.
That is why we break down how automatic trading bot restart covers each of these scenarios individually — from process liveness monitoring to state reconciliation with the exchange before returning to trading.
What automatic trading bot restart provides
A trading bot stops not because of the strategy, but because of the environment: interruption of the quote stream, an exception in the order handler, memory exhaustion, or a hung loop.
While the process is down, the open position remains unattended, and a manual restart at night means dozens of minutes of downtime that no one measures. That is exactly why downtime becomes a working metric just like profitability.
Automatic trading bot restart addresses this at the environment level. A supervisor — for example, systemd — keeps the process under observation, brings it up on crash, and returns it to service in seconds instead of minutes.
In a container setup, restart policies play the same role, while loop liveness checks distinguish a truly running process from a formally started one.
Recovery time stops depending on who is on duty. On startup, the bot pulls current positions, reconciles them with the trade log, and continues from the same point rather than from scratch.
Recovery becomes predictable: the same day and night, on connection loss and on failure of the process itself.
State monitoring and incident notifications close the loop. Liveness metrics, processing latency, and restart counts go to the monitoring system, and an alert arrives only when the loop has not recovered on its own.
This removes an entire class of failures — "the bot crashed, and we learned about it in the morning from a loss" — and manual interventions during night hours are reduced to zero.
Options for organizing restart and their limitations
The choice of option is determined not by convenience, but by which failure needs to be covered: process crash, logic hang while the process is alive, or loss of connection to the exchange. These are different classes of failures — and not every scheme distinguishes them.
| Option | When it fits | Limitations |
|---|---|---|
| Service manager | Single bot on a dedicated server, without containers | Sees only process exit: a hung but alive bot continues to hang. Does not account for crash cause or increasing pause between attempts |
| Container and orchestration | Multiple bots, dependency isolation, reproducible launch | Restart zeroes in-memory state — open positions, local counters. Liveness checks are written manually, and it does not see business logic |
| Watchdog process | Need to catch not "did it die" but "is it working": freshness of trades and quotes, presence of connection | Requires maintenance: with incorrect conditions it spawns bot copies, and the conditions themselves have to be revised when the strategy changes |
| External monitoring | Need metrics, failure history, and notifications to the responsible person | Does not heal by itself: you learn about downtime faster, but downtime lasts until the on-call engineer intervenes |
| Manual control | Debugging, strategies with manual entry confirmation | Reaction time — minutes and hours; night, weekends, and vacation remain uncovered |
Working schemes are almost always combined: a service manager or container provides basic restart, a watchdog process adds logic checks, and external monitoring adds notifications and history. Next, we break down how to assemble these layers into a single loop with predictable behavior.
Restart policies, dependencies, and bot state
The restart policy is described explicitly in configuration, not "just in case." We set the delay between attempts, the retry limit, and the behavior after it is exhausted: the systemd supervisor restarts the process on failure, but on looping it stops it and sends an event to the notification system instead of an endless cycle.
The dependency startup order is more important than the policy itself.
The bot starts after the position store, task queue, and confirmed exchange access, and each step is confirmed by a readiness check rather than by the fact that "the process is started." Otherwise, restart begins on an empty cache — some orders remain uncontrolled.
Position state lives outside the process: trade log, open orders, position size, and time of last synchronization.
On startup, the bot restores the picture from storage and reconciles it with the exchange, and discrepancies go into a report for the on-call engineer rather than into market actions.
Idempotency of operations covers the main risk — duplicate trades. Each order is assigned its own identifier, so a retry after restart is recognized as already executed and does not turn into a second trade.
Stages of implementing auto-restart and alerts
Auto-restart without an audit of causes is dangerous: the bot starts and crashes in a loop, while the real problem — memory leak, lost connection to the exchange, rejected order — remains unnoticed.
That is why we proceed sequentially: first we analyze why the process crashes, then we design recovery, and only then do we enable automation.
-
Failure audit.
We analyze logs and exit codes and separate external failures from logic errors: how many crashes occurred over the period, which ones repeat, what happens to open positions at the moment of stop — whether they are closed, whether orders are hanging on the exchange.
-
Recovery scheme. We decide what to do immediately after startup: reconcile actual positions with the state in storage, cancel unfilled orders, and repeat only safe operations.
We set an attempt limit and increasing pause between them so the bot does not flood the exchange with requests.
-
Restart configuration. We describe the service as a managed systemd unit with a startup policy and restart rate limiting; the dependency startup order — state storage, queues — is set explicitly. For a container, the same rules live in the restart policy.
-
Alert configuration. We divide signals by importance: process crash, missing liveness signal, series of rejected orders, balance discrepancy.
Each gets its own threshold and channel, with deduplication so the on-call engineer receives one notification per incident rather than a stream.
-
Failure testing. On a test environment, we deliberately crash the process, interrupt the connection, delay exchange responses, and see whether the bot recovers without losing positions. Separately, we clean up false positives: an alert that cries wolf stops being read.
-
Handover to operations. We provide an on-call procedure: typical failures, what to look at in logs, how to start manually, how to act on position discrepancy. We run through scenarios together with your team — on-call duty should not depend on us.
What remains with the client after implementation
After implementation, the client is left not with a set of scattered scripts, but with a working loop with clear intervention points.
All solutions are handed over as a complete package: configurations, diagrams, and procedures that the team can use to maintain the bot independently — or hand over only on-call duty to us.
- Service configurations. systemd files with automatic restart policies, delay between attempts, and a limit on the number of crashes — the bot's behavior on failure is the same regardless of who is on shift.
- Container manifests and policies. For a Docker deployment, container restart and liveness checks are configured: the bot starts again not only after a process crash, but also when it has hung and the connection to the exchange is not working.
- Dependency scheme. What must be available before startup — trading gateway, key storage, database, notification channel — with startup order and wait timeouts.
- Alert rules. Event separation: what goes to the on-call engineer immediately, what accumulates in a report, and what is written only to the log. Thresholds are set so that nightly restarts do not turn into a stream of false calls.
- Response procedure. For each alert type it describes: what the on-call engineer checks, when to restart manually, when to roll back configuration, when to halt trading and escalate.
- Documentation. Instructions for "how to bring up the loop from scratch," secret placement, and the procedure for verifying that the bot is actually trading rather than merely listed as started.
- Journals and logs. Configured rotation and retention period, recording of position state and last requests before restart — using them, the cause of failure is analyzed without guesswork.
Acceptance is scenario-based: we kill the process, cut the network, revoke gateway access, and see whether the bot recovers according to the procedure. Describe the task — we will come back with an estimate and work plan.
Case: the bot stopped requiring night on-call duty
The strategy ran around the clock on several exchanges, and open positions could not be left unattended.
Before implementation, the bot crashed 5–6 times a week: loss of connection to the exchange, unexpected API response format, thread pool exhaustion after a series of reconnects.
Each failure meant a night wake-up, manual restart, and position reconciliation: 40–50 minutes of downtime and three to four disrupted nights a week.
What we changed
The process was moved under a service manager with restart by exit code, and the liveness check looks not at "process started" but at the last successful communication with the exchange.
Before restart, positions and orders are flushed to disk, so a repeated launch does not open duplicate trades. Rate limiting — no more than three restarts per ten minutes — prevents falling into a storm cycle.
Alerts were routed to Prometheus with criticality separation: a single failure is silenced and remains in the log, and the on-call engineer is only woken if three consecutive attempts did not produce a live state.
After implementation, 5–6 crashes per week turned into 1–2 per month, recovery time dropped from 40–50 minutes to 10–20 seconds, and about 90% of failures are closed without a human.
Night wake-ups — 1–2 times per quarter instead of three to four per week, and on-call duty stopped being around the clock.
The restart log also showed the root cause — frequent timeouts during reconnection; they were closed by configuring retries in the API access module. Automation remained a safeguard for a real failure rather than an everyday tool.
How process restart differs from state recovery
Process restart is a technical operation that systemd or Docker can do: a crashed process comes up in seconds and responds to signals again.
But this answers only the question "is the process alive," not the question "is the bot continuing the same trading." Between these states lies the entire difference in the cost of an error.
State recovery is a separate task. Open positions, placed orders, unfinished cycles, and the point at which the calculation was interrupted live in RAM; after a crash, its contents are lost.
The process starts from a clean slate and by default does not know that half the work has already been done.
Hence the first mandatory action on startup — position reconciliation. The bot polls the exchange interface and compares actual positions and active orders with its own model.
A discrepancy means that the previous cycle did not complete fully, and trading cannot resume until the state is brought to a common denominator.
The second loop is the operations log, where intent is written before sending an order and the result after.
It provides the basis for handling duplicates: an order sent before the crash may have already been executed, so a retry is sent with a unique identifier that the exchange will reject as a duplicate. Without the log, recovery turns into guesswork and doubled positions.
The practical order is this: start the process, read the log, reconcile positions with the exchange, cancel duplicates, and only then return the trading loop. This way restart remains a cheap operation, and state recovery remains controlled.
Technical concerns before starting work
Before starting automatic bot restart, clients ask the same engineering questions: will the bot start duplicating orders, what happens if the exchange is unavailable, where access keys are stored. Below are brief answers on how this is arranged in our scheme.
How do you distinguish a false trigger from a real crash?
The supervisor looks not only at the process but also at liveness indicators: time of the last request to the exchange, growth of errors in the log. A restart occurs after a series of failed checks in a row; a single timeout does not trigger it.
Will duplicate orders appear after restart?
Before opening positions, the bot reconciles state with the exchange: active orders and positions are requested anew, and its own records are marked with a client identifier.
Unfinished operations are confirmed or cancelled before the trading logic starts — a second order is not created.
What happens if the exchange fails?
Exchange unavailability is not considered a bot crash: restart does not help here, so a pause with increasing delay between attempts is enabled. When the connection returns, the bot reconciles positions again and continues from the confirmed state.
Where are exchange access keys stored?
Keys are stored not in code or in the repository, but in a secure secret store accessible only to the service account. They do not get into logs or notifications, and we restrict key permissions to trading without withdrawals.
Let's discuss your task and come back with a plan
For the estimate to be accurate rather than "approximate," we need to understand the current launch scheme and the nature of crashes. Describe how the bot starts today: manually in a terminal, via a systemd unit, in a container, or on a schedule on a rented server.
Separately, it is important how often it crashes and what happens to open positions at that moment — they remain unattended, close by stop-loss, or hang in a half-state.
From the answers grows an implementation plan: where exactly to place automatic trading bot restart, which health checks to write, and where to route alerts so that you learn about a failure before you see drawdown. In response you will receive:
- analysis of the current failure point — process crash, thread hang, loss of connection to the exchange, or API key failure;
- restart scheme: Restart policy in systemd, container restart policies, or a custom watchdog process;
- liveness checks that distinguish "the process is running" from "the bot is trading" — freshness of the last exchange response, presence of active subscriptions, absence of stuck orders;
- alert routing: instant notification on crash, daily summary of restarts, escalation on repeated failures;
- safe shutdown and state recovery — what to do with open positions and how not to double orders after startup;
- logging and log retention so causes can be analyzed, not just the fact of restart;
- protection of access to the server and exchange keys under automatic management.
Describe the task — we will come back with an estimate and implementation plan, with stages and order of actions.