Catalog Parsing for 1C-Bitrix: Automated Filling

The supplier didn't provide a feed or API. The catalog needs to be filled with thousands of items, and HTML pages are the only source. Typical situation: a client wants to transfer products from a competitor's or supplier's website to 1C-Bitrix. Automated parsing is essential — manual entry of 10,000 items would take weeks. We have accumulated experience on 100+ projects: from simple static pages to dynamic SPAs with bot protection. We guarantee stable parser operation and full documentation so you can maintain it with your in-house team.

Parsing is not just downloading HTML. It involves structure analysis, infoblock design, image processing, and deduplication. Errors at the design stage lead to duplicates and data loss. Let's break down the key components.

How to design an infoblock for parsing products?

Before launching the parser, it's critical to design the infoblock. Common mistakes: storing all characteristics in one text field, not linking products to sections, ignoring XML_ID. The correct scheme:

• Infoblock sections (b_iblock_section) — replicate the source category structure
• Element XML_ID — unique identifier from the source (URL or internal ID)
• Properties — a separate property for each characteristic group or a Highload block for variable attributes
• Trade offers (CATALOG_TYPE = 3) — if the source has product variations

Why is preserving XML_ID important?

XML_ID is the unique element identifier for external systems. See 1C-Bitrix documentation for details. Without XML_ID, each re-parse creates duplicates. The platform uses XML_ID as an external key for updating elements. If the source changes URL or ID, we manually update the mapping. We store the history of all XML_IDs in a separate parser_progress table — this allows rolling back changes without losing data.

Configuring category parsing for catalog filling

First, we traverse the source category structure and create a section tree using CIBlockSection::Add(). It's important to preserve the section XML_ID and DEPTH_LEVEL — this ensures that on re-parsing, we don't create duplicates but update existing sections.

Algorithm:

1. Get the list of categories from the source (sitemap.xml or recursive navigation traversal)
2. Build a parent→child dependency graph
3. Create sections from root downward, recording a map source_id → bitrix_section_id
4. Save the mapping in a separate table or file for future runs

Parsing product cards for 1C-Bitrix

After building the structure, we traverse product pages. Each product is created via CIBlockElement::Add() or updated via CIBlockElement::Update() if XML_ID matches.

Required fields when creating an element:

IBLOCK_ID, NAME, XML_ID, CODE (SEO-friendly URL), ACTIVE, IBLOCK_SECTION_ID, DETAIL_TEXT, PREVIEW_TEXT, PROPERTY_VALUES (array of properties) 

Generating CODE is a frequent point of errors. Use CUtil::translit() with the flag for replacing conflicting characters. Append a suffix from XML_ID if transliteration results in duplicates.

How to bypass protection when parsing a catalog?

Most catalogs are paginated with 20–100 products per page. The parser should:

• Determine the total number of pages (from pagination tag or X-Total-Count header)
• Save progress to a file or database — so that on failure it can resume from the needed page
• Observe delays between requests (1–3 seconds)

For JS-rendered catalogs (filtering via AJAX) — headless Chrome via Puppeteer. Sometimes it's easier to find the XHR request to the catalog API in the Network DevTools and directly call it. Puppeteer processes dynamic content 5 times faster than Simple HTML DOM with browser emulation.

A typical mistake is not accounting for bot protection. Many sites use Cloudflare or captcha. The parser should simulate real user behavior: delays, random User-Agent, proxy pool. In particularly difficult cases, we integrate captcha recognition services.