Redis task queue setup for web application

Setting Up Redis for Task Queues

Tasks belong in queues when operation is too long for HTTP request: email sending, PDF generation, image processing, external API sync. Instead of doing this synchronously and holding connection, task goes to Redis queue and worker executes in background. User gets response immediately.

Redis Data Structures for Queues

Redis supports several queue approaches:

List + LPUSH/RPOP (FIFO) — simplest. LPUSH queue task adds to head, RPOP queue takes from tail. Problem: worker polling empty queue wastes CPU.

BLPOP (Blocking List Pop) — RPOP with blocking: worker waits for task without polling:

BLPOP queue1 queue2 0  # 0 = wait forever
# Returns [queue_name, value] on task arrival

Sorted Set for delayed tasks — tasks with execution time stored in sorted set with score = timestamp:

ZADD delayed_queue 1704067200 "task_payload"  # Unix timestamp
# Worker periodically checks:
ZRANGEBYSCORE delayed_queue 0 NOW LIMIT 0 10
ZREM delayed_queue "task_payload"

Redis Streams — more powerful with consumer groups, acknowledgments (ACK), replay. Recommended for serious production systems.

Laravel Queue with Redis

Laravel Queue is abstraction over various backends. Redis is one of best: fast, supports priorities, delayed jobs, failed jobs.

config/queue.php:

'default' => env('QUEUE_CONNECTION', 'redis'),

'connections' => [
    'redis' => [
        'driver' => 'redis',
        'connection' => 'queue',
        'queue' => env('REDIS_QUEUE', 'default'),
        'retry_after' => 90,  // Seconds until retry if worker crashed
        'block_for' => 5,     // BLPOP blocking seconds
        'after_commit' => true, // Queue after transaction commit
    ],
],

config/database.php — separate Redis connection for queues:

'queue' => [
    'host' => env('REDIS_QUEUE_HOST', '127.0.0.1'),
    'password' => env('REDIS_QUEUE_PASSWORD'),
    'port' => env('REDIS_QUEUE_PORT', '6379'),
    'database' => env('REDIS_QUEUE_DB', '2'),
    'read_timeout' => 60,
],

Create Job:

<?php

namespace App\Jobs;

use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;

class SendOrderConfirmationEmail implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public int $tries = 3;
    public int $timeout = 60;
    public int $backoff = 30; // Seconds between retries

    public function __construct(
        private readonly int $orderId
    ) {}

    public function handle(OrderRepository $orders, Mailer $mailer): void
    {
        $order = $orders->findWithItems($this->orderId);
        $mailer->to($order->customer_email)
               ->send(new OrderConfirmation($order));
    }

    public function failed(\Throwable $exception): void
    {
        // Notify team about failure
        \Log::error('Order confirmation email failed', [
            'order_id' => $this->orderId,
            'error' => $exception->getMessage(),
        ]);
    }
}

Dispatch to queue:

// Immediately
SendOrderConfirmationEmail::dispatch($order->id);

// With delay (5 minutes)
SendOrderConfirmationEmail::dispatch($order->id)->delay(now()->addMinutes(5));

// To specific queue
SendOrderConfirmationEmail::dispatch($order->id)->onQueue('emails');

// Chain of tasks (execute sequentially)
ProcessImage::withChain([
    new GenerateThumbnails($imageId),
    new SendNotification($userId),
])->dispatch($imageId);

Running Workers

# One worker, 'default' queue
php artisan queue:work redis --queue=default --sleep=3 --tries=3

# Multiple queues with priorities
php artisan queue:work redis --queue=critical,high,default --timeout=60

# Daemon mode (doesn't restart PHP between tasks — faster)
php artisan queue:work --daemon

# After new code deploy — restart workers
php artisan queue:restart

Supervisor for production worker management:

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/myapp/artisan queue:work redis --sleep=3 --tries=3 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=4
redirect_stderr=true
stdout_logfile=/var/log/worker.log
stopwaitsecs=3600

numprocs=4 — four parallel workers. For IO-intensive tasks (email, API calls) can be more. For CPU-intensive (image processing) — no more than CPU cores.

Queue Monitoring

Laravel Horizon — official dashboard for Redis Queue monitoring:

composer require laravel/horizon
php artisan horizon:install

config/horizon.php:

'environments' => [
    'production' => [
        'supervisor-1' => [
            'maxProcesses' => 10,
            'balanceMaxShift' => 1,
            'balanceCooldown' => 3,
            'queue' => ['critical', 'default', 'emails'],
            'balance' => 'auto',  // Auto-distribute processes across queues
            'minProcesses' => 1,
            'tries' => 3,
            'timeout' => 60,
        ],
    ],
],

Run:

php artisan horizon

Horizon automatically manages worker count under load and displays metrics: throughput, failed jobs, wait time.

Failed Jobs

Failed tasks saved in Redis (or DB). View and retry:

# List failed tasks
php artisan queue:failed

# Retry specific task
php artisan queue:retry 5

# Retry all failed
php artisan queue:retry all

# Delete failed
php artisan queue:flush

Timeline

Basic Laravel Queue setup with Redis and Supervisor — 1 working day. Adding Horizon with monitoring and auto-scaling — another half day. Complex task chains with retry logic and failed job alerts — 1–2 days.