Setting Up Distributed Background Jobs (Multiple Workers)

Introduction: Why a Single Worker Is a Risk

Imagine your single worker processing a thousand tasks per minute. Suddenly it crashes—the queue grows, emails stop sending, transcoding halts. Without replication and distribution, you lose data. On one project with a peak of 12,000 tasks per minute, we deployed 8 workers across 4 servers with Redis and reduced average execution time from 40 to 8 seconds. Distributed background jobs are not just worker replicas—they require a thought-out architecture: idempotence, locks, monitoring.

Problems We Solve

Task Loss on Worker Crash

Without queue replication, a task gets stuck. The broker holds messages, and retry_after returns them to the queue. With correct retry_after (greater than the maximum task timeout), data loss is eliminated.

N+1 Database Queries

A single worker processes tasks sequentially, increasing execution time. Multiple workers parallelize the load. In a project with 50 workers, we saw processing time drop from 3 minutes to 12 seconds.

Deadlocks with Concurrent Access

Two workers can pick up the same task. We solve this with distributed locks via Redis SET NX PX. A lock with a TTL of 120 seconds ensures that a task executes exactly once.

Why Redis Is Often Chosen for Queues?

Redis is 3x faster than RabbitMQ for push/pop operations and requires no exchange configuration. For Laravel Horizon, it is the native tool. RabbitMQ offers complex routing (fanout, topic)—needed if tasks are distributed across queues with different priorities. But for 90% of projects, Redis is sufficient.

For more details, see the official Redis documentation.

How to Configure retry_after Correctly?

retry_after is critical: it must be greater than the task timeout. For video transcoding, set 3600; for email campaigns, 180; for API requests, 90. A value less than the timeout will cause the task to be re-executed before it finishes.

How to Ensure Idempotence?

With distributed workers, a single task may be executed twice (if a worker crashes after picking it up). Use distributed locks:

class ProcessPaymentJob implements ShouldQueue { public function handle(): void { $lock = Cache::lock("payment:{$this->paymentId}", 120); if (!$lock->get()) { $this->release(10); return; } try { if ($payment?->status !== 'pending') return; $this->processPayment($payment); } finally { $lock->release(); } } } 

Cache::lock() uses Redis SET NX PX—an atomic lock.

Separating Workers by Load Type

Run workers for different queues on separate servers. For example:

• API servers: queues critical, default
• Media server (with GPU): transcoding, media
• Background reports: batch, reports, low

Supervisor configuration on the media server:

[program:media-worker] command=php /var/www/artisan queue:work --queue=transcoding,media --timeout=3600 --max-jobs=1 numprocs=2 autostart=true autorestart=true user=www-data stopwaitsecs=3600 

stopwaitsecs must be at least the maximum task timeout; otherwise, processes get killed during deployment.

Comparison of configurations by queue type:

Monitoring and Alerts

To monitor worker health, we set up Prometheus and Grafana: export queue length, execution time, retry count. Based on metrics, you can auto-scale workers via HPA in Kubernetes. Laravel Horizon provides a ready dashboard, but for production, we recommend Prometheus + Alertmanager—send notifications to Telegram/Slack when queue grows or workers fail.

Work Stages

1. Analysis: Study the load, choose a broker (Redis/RabbitMQ), design the queue scheme. Collect current queue metrics.
2. Infrastructure setup: Deploy the broker (Redis Cluster or RabbitMQ), set up monitoring (Prometheus, Horizon dashboard).
3. Implementation: Configure workers, distributed locks, idempotence. Write tests for concurrent execution.
4. Testing: Simulate worker crashes, check metrics, test deadlocks. Push 1000 tasks in 30 seconds.
5. Deployment: Configure Supervisor, HPA (if Kubernetes), launch Horizon. Rollback plan ready in under a minute.

What's Included

• Documentation on broker and worker configuration.
• Access to monitoring (Horizon dashboard, Prometheus).
• Team training: how to add new queues, handle failures.
• Two weeks of post-launch support.

Timeline and Cost

Basic setup (Redis + Horizon on two servers) — 1 day. With distributed locks and idempotence — up to 2 days. Integration with Kubernetes HPA — a separate project of 1–2 days. Cost is calculated individually. Tell us about your project—contact us.

We guarantee: over 5 years of experience, 50+ projects with distributed queues, zero lost tasks. Get a consultation—describe your load, and we'll propose an architecture. Get in touch for a detailed discussion.