ETL Process Development for 1C-Bitrix

We often encounter a situation where the standard import via the 1C-Bitrix admin panel no longer handles regular synchronization. ERP, 1C, warehouse systems, marketplaces — each source pulls data in its own format. Without a full-fledged ETL, within a month you find that 3% of products have incorrect stock balances, and no one knows about it. We develop turnkey ETL processes: with transformation, error handling, and monitoring, so you can sleep soundly. Contact us for a preliminary assessment of your project.

How Does an ETL Process Solve the Data Inconsistency Problem?

ETL (Extract, Transform, Load) based on 1C-Bitrix is built around three layers. Extract — obtaining data from sources: files (CSV, XML, JSON, YML) via FTP/SFTP/HTTP, REST APIs of external systems (1C, SAP, Salesforce), direct database connections (MySQL, MSSQL, PostgreSQL) via PDO, message queues (RabbitMQ, Kafka). Transform — converting data to Bitrix structure: field mapping, format normalization, validation. Load — writing to Bitrix via D7 API or direct SQL queries for high volumes. This approach ensures data is always up-to-date and aligned with business logic.

Product Load Architecture

For loading products via the standard Bitrix API, we use \Bitrix\Iblock\ElementTable and CCatalogProduct. For volumes from 10,000 products, key settings:

// Отключаем ненужные обработчики на время импорта
define('STOP_STATISTICS', true);
define('NO_AGENT_STATISTIC', 'Y');
define('DisableEventsCheck', true);

// Отключаем поисковый индекс — перестроим в конце
\CSearch::DisableIndex();

// Загрузка элемента инфоблока
$el = new \CIBlockElement();
$result = $el->Add([
    'IBLOCK_ID' => CATALOG_IBLOCK_ID,
    'NAME' => $item['name'],
    'CODE' => $item['code'],
    'ACTIVE' => 'Y',
    'PROPERTY_VALUES' => [
        'VENDOR_CODE' => $item['vendor_code'],
        'WEIGHT' => $item['weight'],
    ],
]);

For volumes > 50,000 items, direct calls to CIBlockElement::Add degrade due to event cascade. We switch to direct INSERTs into tables b_iblock_element, b_iblock_element_property, b_catalog_product with full index rebuild afterward. This yields 3-5x performance improvement.

Why Incremental Synchronization Is Critical

Full overwrite every N hours is expensive and server-intensive. Incremental ETL processes only changed records:

// Фиксируем момент начала синхронизации
$syncStartTime = new \Bitrix\Main\Type\DateTime();

// Запрашиваем из источника только изменённые с прошлой синхронизации
$changedItems = $source->getChangedSince($this->getLastSyncTime());

// После успешной синхронизации обновляем метку
$this->setLastSyncTime($syncStartTime);

Table for storing sync state:

Data Transformation: Typical Challenges

Transformation is the most complex part. Category mapping: source may have a flat list with parent_id, Bitrix uses a tree of sections. We build a tree, map by code or external_id. Price normalization: source gives price with VAT, without VAT, in different currencies. Recalculate using exchange rates. HTML cleanup: descriptions from 1C often contain unreadable formatting — run through DOMDocument, remove unwanted tags. Deduplication: if source does not guarantee unique SKUs, implement merge logic for duplicates.

Row-Level Error Handling

ETL should not stop due to one invalid record:

foreach ($items as $item) {
    try {
        $transformed = $this->transform($item);
        $this->load($transformed);
        $this->stats->incrementSuccess();
    } catch (\Bitrix\Main\ArgumentException $e) {
        // Ошибка валидации — логируем и продолжаем
        $this->logger->warning('Validation failed', [
            'external_id' => $item['id'],
            'error' => $e->getMessage(),
        ]);
        $this->stats->incrementError($item['id'], $e->getMessage());
    } catch (\Exception $e) {
        // Неожиданная ошибка — логируем, но продолжаем
        $this->logger->error('Load failed', ['item' => $item['id'], 'error' => $e->getMessage()]);
        $this->stats->incrementError($item['id'], $e->getMessage());
    }
}

After sync, a report shows how many created, updated, skipped with errors. If errors > 5%, alert is triggered.

Memory Management for Large Volumes

PHP easily exhausts memory when processing 100,000 records. Rules:

• Read data in chunks, do not load entire file into array
• Use generators for CSV/XML iteration
• Explicitly call unset() after processing a chunk
• Reset Bitrix ORM cache: \Bitrix\Main\ORM\Data\DataManager::cleanCache()
• Monitor memory_get_usage() — log if approaching limit

// Генератор для чтения большого CSV
function readCsvChunks(string $file, int $chunkSize = 500): \Generator {
    $handle = fopen($file, 'r');
    $header = fgetcsv($handle);
    $chunk = [];
    while (($row = fgetcsv($handle)) !== false) {
        $chunk[] = array_combine($header, $row);
        if (count($chunk) >= $chunkSize) {
            yield $chunk;
            $chunk = [];
        }
    }
    if ($chunk) yield $chunk;
    fclose($handle);
}

Agents vs Cron vs Queue

• Bitrix agents (b_agent) — for small tasks (up to 1000 records per run). Unstable under low traffic.
• Cron — more reliable for regular syncs: */30 * * * * php -f /var/www/bitrix/etl/sync_products.php >> /var/log/etl.log 2>&1
• Queue (RabbitMQ/Redis) — for event-driven ETL when source publishes change events. Allows handling high-frequency changes without losses.

ETL Monitoring

What's Included

• Source analysis: data structure, formats, schedule
• Extract connector development for each source
• Transformation: mapping, normalization, validation
• Load into Bitrix: API or direct SQL, performance optimization
• Error handling and monitoring: logging, alerts, reports
• Documentation: ETL process description, admin guide
• Training: knowledge transfer to your team
• Post-launch support: stable operation guarantee

Development Stages

Total: 6–12 weeks depending on complexity. Contact us to discuss your task. Order ETL process development and get an engineer consultation.

More about synchronization implementation via CommerceML.