Trading bot monitoring: healthcheck, alerts, watchdog in production

Why uptime monitoring of a trading bot fails in production

Uptime monitoring is usually built around the process: whether the container is alive, whether the port responds, whether the log file is fresh. This catches a complete crash, but does not answer the main question — is the bot trading.

A bot availability check passes green when the process is running while the trading loop is stalled.

A stalled trading loop looks like normal operation: the thread is alive, memory is normal, there are no errors. The loop is waiting for an exchange response and not getting it — the timeout is not set or is set too generously. While the wait drags on, signals are not processed, and the order queue grows.

The second point — loss of connection to the exchange without an explicit error. The socket is open, the market data stream is silent, the logs are empty, process metrics are flat.

This is a silent failure: a port check will not distinguish it from normal silence in a calm market, and the last quote could have arrived tens of minutes ago.

The third point — an accumulated order queue. After connectivity is restored, the bot sends everything it has accumulated at once, and instead of one position it opens a series, some of which are no longer needed.

Duplicate placements and position accounting discrepancies cost more than the downtime itself. Downtime in production is measured not by unavailability time, but by open positions and deviation from the expected result.

The cause is systemic: uptime is measured by process, not by business signals — trade freshness, exchange response time, loop progress.

External checks over an independent channel are needed because a failure inside the process cannot report itself: watchdog and alerts must come from outside. Next, we break down how this is closed.

Effect of implementation: response, downtime, and failure classes

Before monitoring was implemented, a trading bot failure was discovered after the fact — by missed signals, position discrepancies with the exchange, or manual reconciliation.

Detection took from tens of minutes to several hours, and all that time the bot either was not trading or held open positions without control.

Collecting metrics, liveness checks, and alerts on Prometheus turn this work into measurable values, rather than "it seems to have become calmer."

• Incident detection time: from tens of minutes to seconds — the signal comes by threshold, not after reviewing trades; the difference is tens of times.
• Downtime duration: auto-restart and protective mode remove most of the unavailability window; manually, only failures requiring a human decision remain.
• Eliminated failure classes: stuck exchange connections, order desynchronization, and silent trading loop stops are caught by the absence of expected events, not by a missed trade.
• Trading loop stability: the share of correctly closed loops becomes measurable, stays at the target level, and deviations are visible before position discrepancies accumulate.

Each indicator goes into the on-call report: you can see where monitoring triggered and where the threshold was set too late. Thresholds and rules are adjusted based on facts, so response does not degrade over time.

Uptime monitoring methods: external polling, agent, log analysis

None of the three methods gives the full picture, so in production they are combined.

External availability checks answer the question "is the bot visible from outside," application telemetry — "what is happening inside," log analysis — "why this happened." Confusing areas of responsibility is costly: an entry point check will not notice that the bot stopped placing orders, and an internal metric will stay silent if the process hangs together with the node.

A separate requirement is an independent observation point.

The check must come from a node that does not share network, power, and credentials with the trading server: otherwise an incident at the site takes down both the observer and the observed, and the signal comes from the client, not from the system.

The practical minimum is an external check plus telemetry; logs are connected when there is a need to quickly analyze causes. How this is assembled into a single loop with thresholds and notifications — below.

Implementation order: from failure point audit to handover

Implementing monitoring is not installing a single watchdog process, but six steps, each verified in practice. The order is the same for the trading core and for related services: a failure must be noticed before it turns into losses.

1. Failure point audit. We build a map: trading core, connection to exchange interfaces, order storage, notification channel. For each node, we record the signal by which it is clear that it has stopped producing results.
2. Check scheme. We decide what is collected externally and what from inside the process: interface availability, discrepancy between local state and exchange state, queue processing latency, and stuck orders.
3. Alert thresholds. We set them for degradation, not only for crashes: silence during an active market, balance discrepancy beyond tolerance, growth in response time. The threshold triggers before the discrepancy becomes irreversible.
4. Training failure. We artificially cut connectivity and stop part of the services to see the whole chain. We make sure the signal reaches the on-call engineer and does not settle in logs.
5. Response procedure. We describe who receives the notification, who stops the bot, and what counts as a false trigger. Without this, a configured system leads to disputes at the moment of an incident.
6. Handover to operations. We hand over dashboards with statuses, documentation on thresholds, and a checklist for each alert type, and train the on-call shift. We review thresholds together with you: the market and load change, and noise in alerts is the main reason monitoring stops being read.

Output artifacts: configurations, schemes, procedures

After the loop is delivered, the customer is left with a set, not a scattering of scripts on a server: from it, an on-call engineer or another team can deploy monitoring from scratch and maintain it without us.

Everything is stored in a repository, changes go through review and are rolled out like a regular release, rather than edited by hand in prod.

• Check configuration — declarative files with bot service addresses, polling intervals, response timeout limits, and degradation thresholds for each indicator. A new check is added with one entry, without a rebuild.
• Alert rules — routing by severity: critical raises the on-call engineer, degradation goes to the working channel. Duplicate suppression and grouping during a mass failure are described separately, so the shift does not drown in a hundred identical messages.
• Service relationship scheme — a dependency graph from the trading logic entry point to the exchange connection, order database, and task queue. The scheme marks failure points and which nodes degrade together with each of them.
• On-call procedure — responsibility windows, first response time by each severity, escalation order, and criteria for switching the bot to manual trading mode.
• Diagnostics instructions — for each alert: what to look at first, where to check state, how to distinguish a network failure from a strategy logic error, and what to attach to a ticket.
• Watchdog process rules — the limit of automatic restarts, the pause between them, and the condition for a full stop on a recurring failure, so the restart loop does not cut open positions.

Case study: how night downtime of a trading bot was eliminated

The bot traded on three venues at once and placed orders based on strategy signals. Monitoring amounted to logs on the server, which were read in the morning — after the night session had already run.

The review showed the price of such a scheme: six failures in the billing month, five of them found only by market open, average detection time — about seven hours.

We moved outside the signals by which the bot's liveness is visible: a regular heartbeat, freshness of the last exchange API response, discrepancy between the position in the strategy and the actual account.

The watchdog takes down a process that has stopped sending signs of life and brings it back up, while the on-call engineer receives an alert with context — what exactly failed and how long it has lasted.

The monitoring dashboard on Grafana keeps history, so incident analysis is based on facts, not stories.

Metrics before and after

The share of downtime fell not only due to fast detection — the watchdog closes part of the failures itself, without a human.

Previously, every morning began with manually checking open positions and reconciling orders; now the on-call engineer sees the bot's state on one dashboard and handles only those incidents that truly require a decision.

How does a watchdog differ from a regular metric alert?

A threshold alert answers the question "has the metric gone beyond the norm?" and hands the decision to a human.

A watchdog mechanism answers differently — "what should be done right now to keep the bot running" — and does it itself, without waiting for the on-call engineer.

A threshold alert looks at values: latency between signal and order submission, loss of connection to the exchange, absence of trades over a time window. When the boundary is crossed, a notification is sent.

The weak point is that the metric is often collected inside the same process that crashed — and the alert is silent exactly when it is needed most.

A watchdog mechanism is a separate loop that keeps a healthcheck outside the trading core.

It checks not only "the process is alive," but also signs of degradation: whether the quote stream is stuck, whether time from signal to execution is growing, whether the network node responds.

Its resilience lies in the fact that the loop is not tied to the bot's infrastructure — otherwise it goes down together with it.

Next, a response scenario described in advance kicks in: soft reconnect, then worker restart, then stopping trading and moving positions to a safe state.

A safeguard is mandatory — an attempt counter and a window between them, otherwise auto-recovery goes into an endless restart loop and finishes off what could still have been saved.

An alert informs; a watchdog mechanism maintains uptime. In production, both are needed: the first shows where the weak point in the strategy is, the second closes the window while no one is on shift.

Technical questions: false triggers, windows, access

Before launch, the engineering team asks the same questions: where is the boundary between a hang and a deliberate pause in trading, how not to drown in false triggers, will it be necessary to touch an already running bot. Below are short answers to each doubt, without generalities.

Uptime monitoring is built to separate "silent because it is waiting for a signal" and "silent because it crashed." Individually, neither heartbeat nor exchange response latency shows this — only their combined picture does.

What to do with an already deployed bot — will it have to be restarted?

No. Metrics and logs are collected from the side, without interfering with the trading loop. The state reporting point is added in the nearest planned update.

Is access to the trading account needed?

For observation — no, read permissions for balance and order history are enough. Access levels are separated: the on-call engineer sees status and alerts; only the responsible person can change the configuration.

How to remove false triggers?

Each rule goes through a silence window: the signal is sent only if the deviation persists longer than the specified interval. A spike in exchange latency or a brief network degradation does not wake the on-call engineer.

Who handles a night alert?

On-call with escalation: if there is no response within the allotted time, a second engineer joins. Criteria for a trading pause are fixed in advance, so the decision is made according to the procedure.

Let's discuss the task: we will assemble a check scheme and provide a work plan

Conversation starts not with an estimate, but with input data — without them, the check scheme will be guessing. Below is what is worth preparing for the discussion, and what we will return in response.

The timeline estimate comes together with the work plan, not as a separate number: you can see how long each step takes and what appears in production after each one.

• How the bot is launched: one instance or several, whether there is a standby one, and how traffic switches when the primary fails.
• Critical scenarios: placing and canceling orders, behavior during exchange interface failures, reaction to loss of connectivity and to stale quote data.
• Logs and failure history for recent weeks — they show which checks will give a signal before the customer notices the problem.
• Where and to whom the notification goes: channels, on-call engineers, night time, the "wake up or wait" threshold.
• Stopping requirements: should the watchdog process close positions itself or only notify.
• Access restrictions: what can be read in production and what can be changed, and who confirms changes.
• Trading mode: around the clock or in sessions — polling frequency and the allowable downtime window depend on this.

Uptime monitoring is only part of the picture here: a bot can be alive by process and still not trade, so the scheme includes result checks, not only presence.

Based on the discussion, we return with a list of metrics, thresholds, polling frequency, notification route, and escalation order. We attach a staged work plan and a timeline estimate for each item to the scheme.

Something from the list may be missing — that will become clear during the conversation, and our engineers close some of the gaps. Describe the task, and we will return with an estimate and plan.