Background Jobs Setup: Sidekiq, Celery, BullMQ

Imagine: your HTTP request not only returns a response but also sends emails, generates PDFs, synchronizes with external APIs. Response time grows, client times out. The solution is to offload these tasks to background queues. Our experience implementing Sidekiq, Celery, and BullMQ shows this reduces response time up to 5x, eliminates data loss during failures, and saves up to 60% of server resources.

In one Django project, we replaced synchronous email sends with Celery. With 10,000 subscribers, response time dropped from 30 seconds to 200 ms, and database load decreased 4 times. Maintenance cost halved due to lower CPU and memory consumption. Such results are typical for properly configured background processing.

How to choose between Sidekiq, Celery, and BullMQ?

The choice depends on your stack and reliability requirements. Sidekiq is the standard for Ruby/Rails, runs on Redis, supports retries and scheduler. Celery is a universal broker for Python (supports Redis, RabbitMQ, SQS). BullMQ is a modern Node.js orchestrator with built-in repeat tasks. We compared them by key parameters:

BullMQ processes small tasks 20% faster than Celery due to its async engine, but for complex ETL processes, Celery with Redis remains a reliable choice. Sidekiq is the best option for Rails ecosystem.

What's included in queue setup?

We prepare a complete package:

• Broker configuration (Redis, RabbitMQ, SQS) with memory and persistence optimization.
• Worker code with proper error handling, retry policies (exponential backoff, max_retries), and idempotency.
• Monitoring (Sidekiq Web, Flower, BullBoard) with basic authentication.
• Alerting (integration with Sentry, Telegram) on queue failures.
• Documentation for launch and scaling.
• Team training: how to add a new task, how to debug workers.

All solutions are load-tested — we guarantee the queue won't lose a message or crash the server.

How we do it: example with Celery

Recently, we migrated a Django project from cron jobs to Celery + Redis. Originally, digest emails caused timeouts with 1,000 users. After implementing Celery with celery-beat and Flower:

# celery.py
from celery import Celery
from celery.schedules import crontab

app = Celery('myapp')
app.config_from_object('django.conf:settings', namespace='CELERY')

# settings.py
CELERY_BROKER_URL = 'redis://redis:6379/0'
CELERY_RESULT_BACKEND = 'redis://redis:6379/1'
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_EXPIRES = 3600
CELERY_WORKER_PREFETCH_MULTIPLIER = 1

CELERY_BEAT_SCHEDULE = {
    'send-daily-digest': {
        'task': 'myapp.tasks.send_daily_digest',
        'schedule': crontab(hour=9, minute=0),
    },
    'cleanup-tokens': {
        'task': 'myapp.tasks.cleanup_expired_tokens',
        'schedule': crontab(minute=0),
    },
}

Result: digest generation time dropped from 120 seconds to 3 seconds (user doesn't wait). Database load decreased 4 times due to batch operations in the worker. Official Celery documentation recommends using prefetch_multiplier=1 to guarantee even distribution.

Process

1. Audit — analyze current architecture, load, bottlenecks.
2. Design — select queue, priority scheme, retry policies.
3. Development — write workers, set up monitoring, alerting.
4. Testing — load testing (10,000+ tasks) and recovery from failures.
5. Deployment — CI/CD, containerization (Docker), documentation.

Timeline and cost

Basic setup of one queue with one worker: from 1 day. Full project with scheduler, monitoring, and alerting: 3–4 days. Cost is calculated individually after audit. Order a consultation — we'll assess your project.

Why idempotency is key to reliability?

Re-executing a task due to failure can lead to double charges or duplicates. We implement idempotency: each task has a unique identifier, and the worker checks state before execution. This eliminates double processing even with retries. In our practice — 0 data duplication incidents on 50+ projects.

Typical mistakes when implementing

• No idempotency — a retried task breaks business logic.
• Too many retries — queue gets clogged with dead messages.
• Ignoring monitoring — a task fails and you find out a week later.
• Synchronous calls in the worker — block Eventlet.

We guarantee your background job stack will run stably and scale without surprises. Accumulated experience: 8+ years of implementations.

Setting up Sidekiq (Ruby/Rails)

Sidekiq uses Redis as the queue storage, supports retries, dead tasks, and scheduler.

# Gemfile
gem 'sidekiq', '~> 7.0'
gem 'sidekiq-scheduler'

# config/sidekiq.yml
:concurrency: 10
:queues:
  - [critical, 5]
  - [default, 3]
  - [mailers, 2]
  - [low, 1]

# app/workers/email_worker.rb
class EmailWorker
  include Sidekiq::Job
  sidekiq_options queue: :mailers, retry: 3, backtrace: true

  def perform(user_id, template, variables = {})
    user = User.find(user_id)
    UserMailer.send(template, user, variables).deliver_now
  end
end

# Call
EmailWorker.perform_async(user.id, :welcome)
EmailWorker.perform_in(5.minutes, user.id, :follow_up)
EmailWorker.perform_at(1.day.from_now, user.id, :follow_up)

Setting up Celery (Python/Django)

# tasks.py
from celery import shared_task
from myapp.models import User
from myapp.services import send_email

@shared_task(
    bind=True,
    max_retries=3,
    default_retry_delay=60,
    queue='emails',
)
def send_welcome_email(self, user_id: int) -> dict:
    try:
        user = User.objects.get(pk=user_id)
        send_email(user.email, 'welcome', {'name': user.first_name})
        return {'status': 'sent', 'user_id': user_id}
    except Exception as exc:
        raise self.retry(exc=exc, countdown=2 ** self.request.retries * 60)

# Call send_welcome_email.delay(user.id)
send_welcome_email.apply_async(args=[user.id], countdown=300)

# docker-compose: Celery workers
celery-worker:
  build: .
  command: celery -A myapp worker --loglevel=info --concurrency=4 -Q emails,default
  depends_on: [redis, db]
  environment: *app_env
celery-beat:
  build: .
  command: celery -A myapp beat --loglevel=info --scheduler django_celery_beat.schedulers:DatabaseScheduler
  depends_on: [redis, db]
celery-flower:
  build: .
  command: celery -A myapp flower --port=5555 --basic-auth=admin:password
  ports:
    - "5555:5555"

Monitoring Sidekiq

# config/routes.rb
require 'sidekiq/web'

authenticate :user, ->(u) { u.admin? } do
  mount Sidekiq::Web => '/sidekiq'
end

Flower for Celery is available on port 5555. Shows tasks, workers, delays, retries.

Monitoring tools comparison

Contact us for an audit of your project — get a consultation on queue selection and setup. Reach out for background job implementation — your server will stop waiting.