Crypto bot monitoring: what breaks without observation
A crypto bot breaks silently. The process remains in the task list, the connection to the exchange drops as early as at night, orders are not placed — but there is no crash, no email, no call. The outage is discovered from an empty trades screen and manual balance reconciliation hours later.
The main cause of bot failures is swallowed exchange interface errors: the response code is not checked everywhere, the exception goes to the log, the file grows, and no one reads it.
Add waiting for a response without a timeout and desync of the local portfolio with real positions — the strategy is sure everything works, but orders do not go out.
The cost of downtime is not a failed service but lost trades: a position without a stop, a missed entry, forced liquidation on a sharp move.
A once-a-day check does not catch this, so crypto bot monitoring is built on continuous metric collection and thresholds, not manual walkthroughs.
You need to monitor the periodic heartbeat from the strategy, the delay between signal and order placement, the share of exchange rejections, response time, and balance discrepancy.
Metrics are collected by Prometheus, dashboards are built in Grafana, alerts go to the on-call engineer when a threshold is exceeded — diagnosing hangs takes minutes instead of hours.
We break down how this is covered: which metrics actually predict failure, where to set thresholds, and how to distinguish a one-off exchange failure from a hung bot.
What observability gives: effect in measurable values
Crypto bot observability is not graphs, but reducing two intervals: the time until we learned about a failure and the time until it stopped causing damage.
Bot metric collection covers the delay between strategy signal and order placement, the share of orders rejected by the exchange, queue depth and order book desync, while Prometheus and Grafana dashboards bring this into one picture.
Time to detection: minutes instead of hours
Alert rules are built on deviation from a baseline, not static thresholds, so a volatility spike does not wake the on-call engineer unnecessarily. Incident response time drops from tens of minutes — when the market itself reported the breakage — to one or two minutes.
Separate checks catch "silent" failures: the process is alive, but orders stopped being placed.
Dashboards: the cause is visible on the chart, not in logs
Panels are layered — exchange connection, strategy, execution, wallet state. The on-call engineer sees not "the bot is not working" but a specific area: the quote stream diverged, the order handler stalled, or the confirmation rate dropped. The same chart shows when the metric returned to normal — recovery is confirmed by data, not words.
Recovery and share of incidents without consequences
When the cause is localized from the graph, it is enough to roll back configuration or restart one node, not the whole bot — recovery time is cut many times over.
Position discrepancies, stuck orders and lost trades no longer survive until a human notices them: they are closed at the deviation-from-normal stage.
Ways to organize crypto bot monitoring: what to choose
Crypto bot monitoring covers two different tasks: knowing the process is alive and trading, and knowing it is doing so correctly. One data source does not provide both pictures, so monitoring schemes differ not by tools but by which failures they can even catch.
An agent on the same server with metric collection via Prometheus gives depth: signal processing delay, order queue length, exchange connection state.
External availability checks answer "is the bot alive" from an independent point, log analysis answers "why did it not work". Below is a comparison by blind spots.
| Monitoring scheme | When it fits | Blind spot |
|---|---|---|
| Agent on the same server | Need process metrics: processing delay, order queue, exchange connection state | Server or network failure — metrics stop arriving together with the bot, and notification does not come |
| External availability checks | Need an independent point confirming the bot responds and accepts commands | Sees availability, but does not distinguish a working bot from one hung waiting |
| Log analysis | Cause analysis: at which step the failure occurred, why the order was not placed | The problem is recorded after the fact, the window between failure and reaction stretches |
| Hybrid scheme | Continuously operating bot with real trades | Requires support: thresholds and notification rules are revisited after every strategy change |
Hybrid is the working scheme for a production bot: the agent collects internal metrics, an external probe checks availability from outside, logs are aggregated separately.
Matching two signals cuts false positives, so notifications arrive when relevant, and the window between failure and reaction shrinks from hours to minutes.
Implementation order: from audit to handover of runbooks
Implementation begins not with deploying the collector, but with a monitoring implementation plan: what already works in the environment, which failures are critical, and how quickly the team must learn about them.
Without this, dashboards draw pretty graphs but do not answer the main question — why the bot stopped executing orders and when it started.
- Inventory of nodes. We create a map of the environment: RPC nodes, signing service, job queues, strategy workers, storage. For each node we record owner, operating mode and dependencies — the entire monitoring scheme is built on this map.
- Monitoring scheme. We separate what is collected from the application and what from infrastructure: node response delay, number of confirmed transactions, gap between signal and execution, state of keys and limits.
- Collector setup. We deploy metric collection and storage on Prometheus and Grafana, set scrape intervals and retention so data volume does not grow uncontrollably. We assemble dashboards by role: on-call, developer, manager.
- Trigger thresholds. We calculate thresholds from the bot's own statistics, not "by eye": warning level and critical. This way a normal spike in network activity does not wake the on-call.
- Alert testing. We run on live failures: kill a node, slow down the queue, revoke access. We see who received the signal, in what time, and whether it was lost in noise.
- Monitoring handover. On-call runbooks, notification channels, instructions for typical failures, roles and access, team training.
Handover kit: configurations, diagrams, on-call runbooks
Crypto bot monitoring outlives the developer only in one case — if it is handed over as a working system, not as a set of dashboards on someone else's account.
That is why the handover kit includes not only panels, but also what is used to reproduce them, be on call, and investigate failures.
- Configuration files — exporters, job definitions, dashboards and alert rules are in version control with meaningful change history. Deployment of the monitoring stand is repeated from the repository, without manual edits on the server.
- Bot node diagram — which processes, on which nodes, what exchanges data with what, where the failure points are: loss of exchange connection, order queue lag, unavailability of the trade history database.
- Alert rules — for each signal, threshold, averaging window, priority and recipient are recorded. "Burning now" is separated from "check in the morning" — otherwise the on-call stops reacting to notifications.
- On-call instructions — procedure from trigger to escalation: what to check, which commands are safe, when to stop trading logic, whom to raise and via which channel.
- Incident review runbook — recording deadlines, mandatory review fields (symptom, root cause, action), criterion under which an alert rule is rewritten rather than muted.
- Access and roles — separate accounts for on-call and administrator, issuance and revocation procedure, so a person leaving does not leave the system unattended.
Handover ends with acceptance: the client's engineer independently raises the stand from the repository, conducts a training trigger and goes through the review according to instructions. If something does not reproduce at this step, we supplement the kit until it does.
Case: bot failures were found from complaints, now — in minutes
Environment before: logs exist, no observability
The bot traded on two exchanges and stored logs in files on the server. Formally, data for analysis existed — in practice no one opened it: thresholds were not calculated, state snapshots were not saved.
The first serious failure looked like this: one leg of the trade executed, the second did not, the position remained hanging open for several hours.
The team learned about the problem from the client the next morning. Incident review from logs took almost four hours: it was necessary to manually match application records, exchange responses and order history.
The bot monitoring case began exactly with this — not with a desire to "make pretty graphs", but with the need to understand what happened without archaeology.
What was changed
The application now exposes metrics for each trade leg: number of open and stuck orders, exchange response delay, balance discrepancy with calculated, task queue depth.
Collection goes via Prometheus, the dashboard is assembled in Grafana, alerts are separated by criticality — balance discrepancy and stuck order call the on-call immediately, latency growth arrives as a warning.
Metrics before and after
- failure detection time: was 3-6 hours (from client complaint), became 30-90 seconds — from a metric alert;
- incident review: was 2-4 hours of manual log matching, became 10-20 minutes: the dashboard shows account and order state at the time of failure;
- stuck positions: closed automatically by rule, the on-call checks only that closing occurred;
- share of failures found before client contact: nine out of ten.
Each incident is now saved with a snapshot of metrics for an hour before and after — the next review starts from a ready picture, not from scratch.
How does external availability checking differ from metric collection?
External availability checking is a black box: an independent probe from a separate environment every few seconds pings the bot's check endpoint and trading API, measuring response time. Essentially it is a liveness signal that does not depend on what the process thinks about itself.
Metric collection works from inside: the bot publishes counters and delays — share of orders rejected by the exchange, order book processing lag, task queue depth, discrepancy of local position with exchange.
Such a loop sees degradation long before failure, but that is exactly why it cannot be considered sufficient.
The main blind spot of internal metrics is the moment when the process hangs or crashes: the numbers simply stop arriving, and this silence is easily confused with stability.
An external probe, on the contrary, catches a hung processing loop, dropped exchange connection, pool exhaustion, network boundary error — but does not explain the cause.
The difference between approaches is that each sees its own thing.
External checking answers whether the service is working right now, while metric collection answers why it became slower and where the trend is heading: order placement lag grows, executed share falls, balance diverges.
Therefore one approach is not enough: external checks without metrics give an alert without diagnosis, metrics without a probe are silent exactly at the moment of breakage.
We set up both loops in a common notification system, with different thresholds and averaging windows, so a signal arrives both for failure and for slow degradation.
Technical doubts: overload, access, false alerts
Before approving monitoring, engineers ask the same questions: will it eat resources, to whom do we give access, will we drown in noise. We address each point.
Will monitoring start competing with trading logic?
Metrics are exposed by a separate endpoint outside the order processing loop: collection runs by interval, not per event, so load does not grow linearly. On the test environment we take a profile before and after enabling monitoring and record the discrepancy in the report.
Will we have to hand over private keys or exchange access?
No. Monitoring reads metrics and logs through a read-only endpoint. Private signing keys never leave the secure environment and do not get into either the collector configuration or dashboards. To control balance, we create a separate key with withdrawal disabled.
How to set thresholds so we do not drown in false alerts?
We do not set thresholds "by eye": first we collect statistics of normal behavior, then set boundaries by percentiles with an averaging window. Triggers are grouped and deduplicated, recurring ones are suppressed — noise does not mix with critical signal. Each alert has an attached action, not "take a look".
What about on-call during non-working hours?
Escalation levels are separated: warnings go to the work channel and are handled in the morning, critical ones raise the on-call. For typical failures — stuck handler, connection loss — we set up auto-recovery, and no one is woken at night. For non-standard cases there is an instruction: what to check, how to roll back, whom to involve.
Let's discuss the task: how to build monitoring for your environment
We start the conversation with the environment: which exchanges and strategies the bot serves, where it runs, who and how learns about a failure today — manually from logs or from notification.
Based on the results, we will return with an estimate and work plan showing which metrics we collect, where we set thresholds, and who receives the signal. Starting bot monitoring usually takes less time than it seems if the inputs are gathered in advance.
What we need from you for an accurate estimate:
- read-only access to the nodes where the bot runs — to the service, configurations and log storage;
- list of processes we monitor: order placement and cancellation, exchange synchronization, balance top-up, stuck tasks;
- list of failures that have already occurred, and your criticality grading — we will use them to prioritize notifications;
- signal delivery channels: messenger, email, call, and who exactly they are addressed to at different times of day;
- maintenance windows and acceptable downtime, so the on-call is not woken by planned work;
- history requirements: how long to store metrics and how often to take snapshots for incident review.
As a result, you get a metric map with a description of each one's meaning, collection configuration, graph panels for key nodes, notification rules with thresholds and anti-flapping protection, as well as a short instruction for the on-call: what to look at first and which actions are safe.
Separately, we will test the scheme on a training failure — stop the handler and make sure the signal reached the right person.
Describe the task — we will return with an estimate and plan.