Automatic Restart of a Trading Bot: systemd, Docker, Alerts

Configuring Automatic Restart of a Trading Bot

A trading bot runs 24/7. A crash due to a network error, OOM, unhandled exception, or system update — and the process dies. Without auto-restart, the bot stays dead until an operator intervenes. The longer the downtime, the more missed profit and risk of missing trading signals. In production, we achieve 99.9% uptime using a combination of systemd, graceful shutdown, and alerts. We'll show how to set this up using a real project example: a Python bot with aiohttp running on Ubuntu 22.04 with 4 cores. We use systemd for management, Docker for isolation, and Prometheus for monitoring.

systemd is the standard service manager on Linux. It can automatically restart a process, limit resources, and log. But simply setting Restart=always isn't enough — you need protection against crash loops. Let's look at three key components: the systemd unit, graceful shutdown in the bot code, and alerts on failures. For containerized bots, we supplement with Docker restart policy and health checks.

How to Configure systemd for Automatic Restart

Create a unit file in /etc/systemd/system/ with Restart=always and RestartSec=10. Additionally set StartLimitBurst=5 and StartLimitIntervalSec=60 to avoid infinite restarts on critical errors. Enable the service: systemctl enable trading-bot.

# /etc/systemd/system/trading-bot.service
[Unit]
Description=Trading Bot
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=botuser
WorkingDirectory=/opt/trading-bot
ExecStart=/opt/trading-bot/venv/bin/python -u bot.py
Restart=always
RestartSec=10
StartLimitIntervalSec=60
StartLimitBurst=5
EnvironmentFile=/opt/trading-bot/.env
MemoryLimit=2G
CPUQuota=80%
StandardOutput=journal
StandardError=journal
SyslogIdentifier=trading-bot

[Install]
WantedBy=multi-user.target

Activation:

systemctl daemon-reload
systemctl enable trading-bot
systemctl start trading-bot
journalctl -u trading-bot -f # live logs

The parameters StartLimitBurst=5 + StartLimitIntervalSec=60 provide protection against crash loops. Without them, a bot that keeps crashing would restart indefinitely, accumulating errors (open positions, duplicate orders). After 5 quick crashes, systemd stops the service and triggers an alert (if configured). This is 5 times more reliable than a simple cron monitor.

systemd Unit Parameters: Detailed Breakdown

These parameters increase stability: the bot doesn't crash from overloads, and on frequent errors, systemd blocks startup, preventing losses due to double orders.

What Is Graceful Shutdown and Why Is It Needed?

You cannot kill a bot with SIGKILL — it may leave open orders, uncommitted positions, unsent alerts. Handle SIGTERM:

import signal
import asyncio

class TradingBot:
    def __init__(self):
        self.running = True
        self.open_orders: list = []

    async def shutdown(self):
        self.running = False
        for order_id in self.open_orders:
            try:
                await self.exchange.cancel_order(order_id)
            except Exception as e:
                logger.error(f"Failed to cancel order {order_id}: {e}")
        logger.info("Graceful shutdown complete")

    async def run(self):
        loop = asyncio.get_event_loop()
        loop.add_signal_handler(
            signal.SIGTERM,
            lambda: asyncio.create_task(self.shutdown())
        )
        while self.running:
            try:
                await self.main_loop()
            except Exception as e:
                logger.exception(f"Error in main loop: {e}")
            await asyncio.sleep(5)

systemd on systemctl stop sends SIGTERM, then after TimeoutStopSec (default 90 sec) sends SIGKILL. For a bot with positions, 90 seconds is usually sufficient.

Docker: An Alternative for Containerized Bots

If the bot runs in Docker, use restart: unless-stopped — it restarts on crash and after host reboot, but not on manual stop. A health check is critical: Docker only restarts a container on a complete crash; a hang without an error goes unnoticed.

# docker-compose.yml
services:
  trading-bot:
    image: trading-bot:latest
    restart: unless-stopped
    env_file: .env
    volumes:
      - ./data:/app/data
      - ./logs:/app/logs
    mem_limit: 2g
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "5"
    healthcheck:
      test: ["CMD", "python", "-c", "import requests; requests.get('http://localhost:8080/health', timeout=5)"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s

Example health endpoint on aiohttp:

from aiohttp import web

async def health_check(request):
    last_loop_age = time.time() - bot.last_loop_time
    if last_loop_age > 300:
        return web.Response(status=503, text=f"Bot stuck: last loop {last_loop_age:.0f}s ago")
    if not bot.exchange_connected:
        return web.Response(status=503, text="Exchange disconnected")
    return web.Response(status=200, text="OK")

app = web.Application()
app.router.add_get('/health', health_check)

Comparison of systemd and Docker for Auto-Restart

Why Alerts on Crash Are Needed

The fact of a restart should generate a notification — even if the bot recovered automatically. The minimal solution is a Telegram bot with the hostname and time. In systemd, this is done via OnFailure=trading-bot-notify.service. For Prometheus: rule changes(process_start_time_seconds{job="trading-bot"}[5m]) > 0. Get a consultation on alert configuration — we'll recommend the optimal solution for your infrastructure.

What’s Included in Auto-Restart Configuration (Deliverables)

• systemd unit or Docker Compose configuration with crash loop protection
• Implementation of graceful shutdown handling open orders
• Health check endpoint verifying bot and exchange state
• Alert configuration (Telegram / Slack) on crashes and frequent restarts
• Testing on a staging environment before production deployment
• Configuration documentation and instructions for your team
• 5+ years of experience in blockchain development — we guarantee stability

Contact us to set up your trading bot — we ensure 24/7 stable operation. Get a consultation on systemd, Docker, and alert configuration.