Crypto Bot Failure Notification Setup

Setting Up Notifications for Crypto Bot Failures

Bot crashed at 3 AM, missed 40 transactions or hung on one operation — you need to know immediately, not 8 hours later when you open laptop. Proper alert system covers three classes of problems: process crash, performance degradation (bot works but slowly or skips events), and business anomalies (no transactions for 2 hours — is that normal or did network crash?).

Typical Crypto Bot Architecture and Failure Points

Most crypto bots are loops: listen to events → process → act. Failure points:

• RPC provider — Infura/Alchemy can return errors, ratelimit, or lag behind block head
• WebSocket connection — breaks, bot continues but doesn't receive events
• Transaction handler — hangs on one, blocks queue
• Insufficient gas/SOL — bot can't send transactions
• Process crashed — OOM, unhandled exception, segfault

Healthcheck Endpoint + External Monitoring

Most reliable approach: bot periodically reports "I'm alive", external service detects silence.

// In bot: heartbeat every 30 seconds
class BotHealthReporter {
  private lastProcessedBlock: number = 0;
  private processedCount: number = 0;

  startHeartbeat(): void {
    setInterval(async () => {
      const payload = {
        status: 'ok',
        lastBlock: this.lastProcessedBlock,
        processed: this.processedCount,
        timestamp: Date.now(),
        rpcLatency: await this.measureRpcLatency(),
      };
      
      // Betterstack/Healthchecks.io ping
      await fetch(process.env.HEALTHCHECK_PING_URL!, {
        method: 'POST',
        body: JSON.stringify(payload),
      }).catch(() => {}); // Don't crash if healthcheck unavailable
    }, 30_000);
  }
}

Healthchecks.io is a free service for deadman's switch alerts. Set up: "if I don't get ping within 2 minutes — send notification". Supports Telegram, Email, Slack, PagerDuty.

Betterstack (ex. Logtail) — analog with richer UI and log integration.

Telegram Bot for Alerts: Faster than Email

class TelegramAlerter {
  private bot: Telegraf;
  private chatId: string;

  constructor() {
    this.bot = new Telegraf(process.env.TELEGRAM_BOT_TOKEN!);
    this.chatId = process.env.TELEGRAM_ALERT_CHAT_ID!;
  }

  async sendAlert(level: 'info' | 'warning' | 'critical', message: string): Promise<void> {
    const emoji = { info: 'ℹ️', warning: '⚠️', critical: '🚨' }[level];
    const text = `${emoji} *[${level.toUpperCase()}]*\n\n${message}\n\n_${new Date().toISOString()}_`;
    
    await this.bot.telegram.sendMessage(this.chatId, text, {
      parse_mode: 'Markdown',
    });
  }

  async sendBlockAlert(currentBlock: number, chainHead: number): Promise<void> {
    const lag = chainHead - currentBlock;
    if (lag > 50) {
      await this.sendAlert('critical', 
        `Bot lagging behind network by ${lag} blocks\nCurrent: ${currentBlock}\nHead: ${chainHead}`
      );
    }
  }
}

Monitoring Chain Head Lag

Bot can work without errors but process blocks with lag — due to slow RPC or overloaded handler. Add lag check:

async function checkChainLag(provider: JsonRpcProvider, lastProcessed: number): Promise<void> {
  const currentHead = await provider.getBlockNumber();
  const lag = currentHead - lastProcessed;
  
  if (lag > 10)  alerter.sendAlert('warning', `Lag: ${lag} blocks`);
  if (lag > 50)  alerter.sendAlert('critical', `Critical lag: ${lag} blocks`);
  
  // Metric for Grafana
  metrics.gauge('bot_chain_lag_blocks', lag);
}

Alert on Low Gas Wallet Balance

async function checkGasBalance(provider: JsonRpcProvider, botAddress: string): Promise<void> {
  const balance = await provider.getBalance(botAddress);
  const balanceEth = parseFloat(formatEther(balance));
  
  if (balanceEth < 0.05) {
    await alerter.sendAlert('warning', 
      `Low gas wallet balance: ${balanceEth.toFixed(4)} ETH\nAddress: ${botAddress}`
    );
  }
  if (balanceEth < 0.01) {
    await alerter.sendAlert('critical', 
      `CRITICAL: gas wallet almost empty: ${balanceEth.toFixed(4)} ETH — bot will stop soon`
    );
  }
}

Process Supervisor: Auto-Restart

If bot crashes — should restart automatically. PM2 for Node.js:

# ecosystem.config.js
module.exports = {
  apps: [{
    name: 'crypto-bot',
    script: 'dist/bot.js',
    restart_delay: 5000,
    max_restarts: 10,
    min_uptime: '10s', // don't count restart if crashed before 10s
    error_file: '/var/log/crypto-bot/error.log',
    out_file: '/var/log/crypto-bot/out.log',
  }]
};

pm2 start ecosystem.config.js
pm2 save # auto-start after server reboot

PM2 itself sends notifications via pm2-notify or keymetrics integration. Each restart — log and send alert via Telegram.

Final Alert Checklist

• No heartbeat > 2 minutes → critical alert
• Chain head lag > 50 blocks → critical alert
• Gas wallet balance < 0.05 ETH → warning
• N errors in a row in handler → critical alert
• Process restarted → info alert
• RPC latency > 5 seconds → warning