Background Job Monitoring: Sidekiq, Bull Board, Flower Dashboards

Background Job Monitoring: Setting Up Sidekiq, Bull Board, Flower Dashboards

A queue without monitoring is a black box: tasks hang, fail, accumulate in the thousands, and you learn about it from users. On one project, we discovered that 30% of tasks in the 'media' queue were dying due to a 60-second timeout — after increasing it to 300 seconds, the success rate rose to 99.5%. Our experience: 50+ implementations for Ruby, Node.js, and Python stacks. We guarantee transparency for your queues.

What Problems Do We Solve?

Typical scenarios: tasks fail with unlogged errors, the queue grows uncontrollably, there are no failure alerts, and debugging stuck tasks is difficult. A dashboard solves this: you see the state of each task, the number of failed attempts, and wait time. For example, on a project with a notification queue, response time dropped by 70% after setting up monitoring and alerting.

How to Choose the Right Dashboard

The choice depends on the stack. For Ruby + Sidekiq — Sidekiq Web UI. For Node.js + BullMQ — Bull Board. For Python + Celery — Flower. Each provides a web interface and REST API. Below is a comparison.

Sidekiq Web UI wins in built-in alerting — it requires 100% fewer external dependencies than Bull Board or Flower.

Why Queue Monitoring Is Critical

Complex distributed systems depend on background jobs. Without monitoring, you don't see where the bottleneck is: in the worker code, queue configuration, or infrastructure. For example, Laravel Horizon's automatic scaling (balance: auto) increases the number of processes as the queue grows — but only if you see the metrics. Implementing monitoring cuts problem detection time from hours to minutes.

Setting Up Laravel Horizon

config/horizon.php defines worker pools. Example configuration with balancing:

'environments' => [ 'production' => [ 'supervisor-1' => [ 'connection' => 'redis', 'queue' => ['high', 'default', 'low'], 'balance' => 'auto', 'minProcesses' => 2, 'maxProcesses' => 10, 'tries' => 3, 'timeout' => 60, ], ], ], 

balance: auto — Horizon automatically scales the number of processes based on queue depth. In production, we start it via Supervisor:

[program:horizon] command=php /var/www/artisan horizon autostart=true autorestart=true user=www-data 

Dashboard authorization is configured via a service provider:

protected function gate(): void { Gate::define('viewHorizon', function ($user) { return in_array($user->email, config('horizon.admin_emails', [])); }); } 

Setting Up Bull Board for Node.js

Install @bull-board/express and bullmq. Example connecting three queues:

import { createBullBoard } from '@bull-board/api'; import { BullMQAdapter } from '@bull-board/api/bullMQAdapter'; import { ExpressAdapter } from '@bull-board/express'; import { Queue } from 'bullmq'; const emailQueue = new Queue('email', { connection }); const serverAdapter = new ExpressAdapter(); serverAdapter.setBasePath('/admin/queues'); createBullBoard({ queues: [new BullMQAdapter(emailQueue)], serverAdapter }); app.use('/admin/queues', authMiddleware, serverAdapter.getRouter()); 

Bull Board shows active, waiting, completed, and failed tasks. From failed, you can manually retry.

Setting Up Flower for Python

Flower runs as a separate service. Using Docker Compose:

flower: image: mher/flower:2.0 command: celery --broker=redis://redis:6379/0 flower --port=5555 environment: - FLOWER_BASIC_AUTH=admin:secretpass ports: - "5555:5555" 

Flower provides a REST API for automation: worker status, task list, task cancellation.

How to Set Up Alerting for Queues

For Horizon, configure waits in config/horizon.php — an alert if a task waits longer than the specified time. Custom integration with Telegram via the LongWaitDetected event (we describe it in code, but omit for brevity). For Bull Board and Flower, alerting is configured separately through external tools like Prometheus + Alertmanager.

'waits' => [ 'redis:default' => 60, ], 

Event::listen(function (LongWaitDetected $event) { // Send notification });