Setting Up Parsing Queues for Large 1C-Bitrix Catalogs

Setting up parsing queues for large 1C-Bitrix catalogs is a challenge faced when dealing with 500,000+ items. Parsing 500 products is a simple script. Parsing 500,000 products is an architecture with queues, workers, monitoring, and a retry strategy. Without queues, such a catalog cannot be processed reliably: the script crashes at the 50,000th item, and no one knows where to resume. We guarantee your system will handle the load and not lose data. Our experience with over 50 projects featuring catalogs of 100,000+ items enables us to design the right architecture from the start. Savings on maintenance due to a stable system amount to substantial annual reductions.

Example: for an electronics e‑commerce store with a 300,000-item catalog, we designed a PostgreSQL queue with 6 workers. Full processing time dropped from 8 hours to 45 minutes. The system has been running stably since launch with no failures.

According to PostgreSQL documentation, using SKIP LOCKED allows atomic task acquisition without deadlock risks.

Why Queue Setup for Large Bitrix Catalogs Is Critical

Execution times exceed limits, scripts crash due to memory limit, and restarts begin from zero. Queues solve these problems: they split processing into atomic tasks, allow resumption from the point of failure, and prioritize important tasks (stock updates, price changes).

Key signs indicating the need for queues: full catalog processing takes more than 30 minutes; scripts periodically fail with timeout or memory limit; it is impossible to determine what has been processed after a crash; different task types with different priorities need to be handled.

How to Build a Reliable Queue in PostgreSQL

A queue table in PostgreSQL is optimal for most Bitrix projects. It is more reliable than Redis (persistent), requires no additional infrastructure, and is convenient to monitor via SQL queries. We use a table with indexes that ensure fast task selection by status and priority.

CREATE TABLE parser_tasks (
    id BIGSERIAL PRIMARY KEY,
    task_type VARCHAR(50) NOT NULL, -- 'parse_product', 'update_price', 'update_stock'
    payload JSONB NOT NULL, -- {'url': '...', 'element_id': 123}
    priority SMALLINT DEFAULT 5, -- 1 (highest) - 10 (lowest)
    status VARCHAR(20) DEFAULT 'pending',
    worker_id VARCHAR(50), -- ID of the worker that grabbed the task
    attempts SMALLINT DEFAULT 0,
    max_attempts SMALLINT DEFAULT 3,
    error_message TEXT,
    available_at TIMESTAMP DEFAULT NOW(), -- for delayed tasks and retry
    created_at TIMESTAMP DEFAULT NOW(),
    processed_at TIMESTAMP
);
CREATE INDEX idx_pt_pick ON parser_tasks(status, priority, available_at) WHERE status IN ('pending', 'failed');

The index on (status, priority, available_at) is critical for fast selection of the next task.

How to Avoid Race Conditions When Grabbing Tasks

Multiple workers must not pick the same task. Atomic grabbing via SELECT ... FOR UPDATE SKIP LOCKED:

UPDATE parser_tasks
SET status = 'processing',
    worker_id = $1,
    attempts = attempts + 1
WHERE id = (
    SELECT id
    FROM parser_tasks
    WHERE status = 'pending'
      AND available_at <= NOW()
    ORDER BY priority ASC, created_at ASC
    LIMIT 1
    FOR UPDATE SKIP LOCKED
)
RETURNING *;

SKIP LOCKED — other workers skip locked rows, avoiding blocks and deadlocks.

How Many Workers Should Be Started?

Each worker is a separate PHP-CLI process. Process management via Supervisor:

[program:parser_worker]
command=/usr/bin/php /var/www/site/local/scripts/parser_worker.php
numprocs=4
autostart=true
autorestart=true
stderr_logfile=/var/log/parser_worker.err.log
stdout_logfile=/var/log/parser_worker.out.log

numprocs=4 — 4 parallel workers. The optimal number depends on CPU, server memory, and the source's rate limit. Rule: the number of workers must not exceed the allowed load on the source (no more than 1 request/sec per domain with 1 worker).

What Does Task Prioritization Give?

Different tasks have different priorities. Priority queues allow critical data to be processed instantly, while background tasks run during idle time. This reduces system response time for stock and price updates.

Stocks and prices never wait in queue behind slow image parsing.

How to Monitor Queue State?

Dashboard in the admin panel:

SELECT task_type, status, COUNT(*) as cnt, AVG(EXTRACT(EPOCH FROM (processed_at - created_at))) as avg_wait_sec
FROM parser_tasks
WHERE created_at > NOW() - INTERVAL '24 hours'
GROUP BY task_type, status;

Alert when the queue accumulates: if pending tasks of type update_price exceed 10,000 — workers are falling behind, need to increase numprocs or slow task generation.

What Our Work Includes?

We offer a full cycle of queue development and setup. The cost is determined after analysis.

1. Designing the queue schema and selecting a strategy (atomic dequeue, retry with backoff)
2. Implementing the worker with handlers for each task type
3. Configuring Supervisor and autoscaling settings
4. Integrating monitoring and alerts (Grafana, Zabbix, or equivalent)
5. Operational documentation and training your team
6. Post-launch support: bug fixes, performance optimization

We can assess your project – contact us to discuss an optimal solution. If you need a stable parsing system, reach out to discuss an audit of your project. Get a consultation to talk details.

Timeline

Total: 4–7 working days — the infrastructure foundation for reliable parsing system operation.