ETL Process Development for 1C-Bitrix

We often encounter a situation where the standard import via the 1C-Bitrix admin panel no longer handles regular synchronization. ERP, 1C, warehouse systems, marketplaces — each source pulls data in its own format. Without a full-fledged ETL, within a month you find that 3% of products have incorrect stock balances, and no one knows about it. We develop turnkey ETL processes: with transformation, error handling, and monitoring, so you can sleep soundly. Contact us for a preliminary assessment of your project.

How Does an ETL Process Solve the Data Inconsistency Problem?

ETL (Extract, Transform, Load) based on 1C-Bitrix is built around three layers. Extract — obtaining data from sources: files (CSV, XML, JSON, YML) via FTP/SFTP/HTTP, REST APIs of external systems (1C, SAP, Salesforce), direct database connections (MySQL, MSSQL, PostgreSQL) via PDO, message queues (RabbitMQ, Kafka). Transform — converting data to Bitrix structure: field mapping, format normalization, validation. Load — writing to Bitrix via D7 API or direct SQL queries for high volumes. This approach ensures data is always up-to-date and aligned with business logic.

Product Load Architecture

For loading products via the standard Bitrix API, we use \Bitrix\Iblock\ElementTable and CCatalogProduct. For volumes from 10,000 products, key settings:

// Отключаем ненужные обработчики на время импорта define('STOP_STATISTICS', true); define('NO_AGENT_STATISTIC', 'Y'); define('DisableEventsCheck', true); // Отключаем поисковый индекс — перестроим в конце \CSearch::DisableIndex(); // Загрузка элемента инфоблока $el = new \CIBlockElement(); $result = $el->Add([ 'IBLOCK_ID' => CATALOG_IBLOCK_ID, 'NAME' => $item['name'], 'CODE' => $item['code'], 'ACTIVE' => 'Y', 'PROPERTY_VALUES' => [ 'VENDOR_CODE' => $item['vendor_code'], 'WEIGHT' => $item['weight'], ], ]); 

For volumes > 50,000 items, direct calls to CIBlockElement::Add degrade due to event cascade. We switch to direct INSERTs into tables b_iblock_element, b_iblock_element_property, b_catalog_product with full index rebuild afterward. This yields 3-5x performance improvement.

Why Incremental Synchronization Is Critical

Full overwrite every N hours is expensive and server-intensive. Incremental ETL processes only changed records:

// Фиксируем момент начала синхронизации $syncStartTime = new \Bitrix\Main\Type\DateTime(); // Запрашиваем из источника только изменённые с прошлой синхронизации $changedItems = $source->getChangedSince($this->getLastSyncTime()); // После успешной синхронизации обновляем метку $this->setLastSyncTime($syncStartTime); 

Table for storing sync state:

Data Transformation: Typical Challenges

Transformation is the most complex part. Category mapping: source may have a flat list with parent_id, Bitrix uses a tree of sections. We build a tree, map by code or external_id. Price normalization: source gives price with VAT, without VAT, in different currencies. Recalculate using exchange rates. HTML cleanup: descriptions from 1C often contain unreadable formatting — run through DOMDocument, remove unwanted tags. Deduplication: if source does not guarantee unique SKUs, implement merge logic for duplicates.

Row-Level Error Handling

ETL should not stop due to one invalid record:

foreach ($items as $item) { try { $transformed = $this->transform($item); $this->load($transformed); $this->stats->incrementSuccess(); } catch (\Bitrix\Main\ArgumentException $e) { // Ошибка валидации — логируем и продолжаем $this->logger->warning('Validation failed', [ 'external_id' => $item['id'], 'error' => $e->getMessage(), ]); $this->stats->incrementError($item['id'], $e->getMessage()); } catch (\Exception $e) { // Неожиданная ошибка — логируем, но продолжаем $this->logger->error('Load failed', ['item' => $item['id'], 'error' => $e->getMessage()]); $this->stats->incrementError($item['id'], $e->getMessage()); } } 

After sync, a report shows how many created, updated, skipped with errors. If errors > 5%, alert is triggered.

Memory Management for Large Volumes

PHP easily exhausts memory when processing 100,000 records. Rules:

• Read data in chunks, do not load entire file into array
• Use generators for CSV/XML iteration
• Explicitly call unset() after processing a chunk
• Reset Bitrix ORM cache: \Bitrix\Main\ORM\Data\DataManager::cleanCache()
• Monitor memory_get_usage() — log if approaching limit

// Генератор для чтения большого CSV function readCsvChunks(string $file, int $chunkSize = 500): \Generator { $handle = fopen($file, 'r'); $header = fgetcsv($handle); $chunk = []; while (($row = fgetcsv($handle)) !== false) { $chunk[] = array_combine($header, $row); if (count($chunk) >= $chunkSize) { yield $chunk; $chunk = []; } } if ($chunk) yield $chunk; fclose($handle); } 

Agents vs Cron vs Queue

• Bitrix agents (b_agent) — for small tasks (up to 1000 records per run). Unstable under low traffic.
• Cron — more reliable for regular syncs: */30 * * * * php -f /var/www/bitrix/etl/sync_products.php >> /var/log/etl.log 2>&1
• Queue (RabbitMQ/Redis) — for event-driven ETL when source publishes change events. Allows handling high-frequency changes without losses.

ETL Monitoring

What's Included

• Source analysis: data structure, formats, schedule
• Extract connector development for each source
• Transformation: mapping, normalization, validation
• Load into Bitrix: API or direct SQL, performance optimization
• Error handling and monitoring: logging, alerts, reports
• Documentation: ETL process description, admin guide
• Training: knowledge transfer to your team
• Post-launch support: stable operation guarantee

Development Stages

Total: 6–12 weeks depending on complexity. Contact us to discuss your task. Order ETL process development and get an engineer consultation.

More about synchronization implementation via CommerceML.