In our practice, an online store on 1C-Bitrix without competitor data quickly loses ground on pricing and assortment. Imagine: 3000 SKUs in the catalog, three direct competitors, manual monitoring taking 20 hours per week. One competitor drops the price on a top model — you learn about it a week later, losing up to 15% of sales. Manual monitoring of 500+ items is unprofitable — automation is required. We offer a parser that collects product data from competitor websites and loads it into a separate Bitrix infoblock for analysis and automated actions. Our basic parser starts at $1,200, and clients typically save $2,000 monthly on manual monitoring. We'll evaluate your project in 1 day — get in touch.
Why Competitor Parsing Is Critical for a Bitrix Online Store?
Without up-to-date information, you risk losing on price or missing new products. The parser tracks changes from 3–5 competitors in real time and generates a deviation summary. This allows faster reaction than competitors — speed of reaction directly impacts conversion. Automated parsing is 5x more efficient than manual checks.
Solution Architecture
The parser is a separate module that does not affect the main catalog. Typical schema:
- Collector — PHP/Python script with Guzzle or Symfony HttpClient, crawling competitor pages.
-
Intermediate storage — a separate infoblock
COMPETITORS_CATALOGor a database table. -
Analytical layer — comparison with
b_catalog_priceof your store. -
Trigger actions — price update via
CCatalogProduct::Update()or manager notification.
Storing competitor data directly in the main catalog is bad practice: it clutters b_iblock_element, breaks search indexes. Better to use a separate infoblock linked via XML_ID. As noted in 1C-Bitrix documentation: storing external data in a separate infoblock ensures main catalog performance.
Technical Challenges and How to Bypass Them
Anti-scraping protection is the main obstacle. Large stores use Cloudflare, dynamic JS loading (React/Vue SPA) and captchas. Against static sites, curl with rotating User-Agent works. Against JS rendering, a headless browser is needed: Puppeteer via Node.js or Playwright. Compared to Selenium, Playwright is 2–3 times more stable and faster.
Stack for JS sites:
Playwright → stdout JSON → PHP reads via exec() → CIBlockElement::Add() Unstable HTML structure — competitor changed layout, parser breaks. Solution: use CSS selectors instead of XPath where structure is flat, and mandatory monitoring with alert on zero result selection.
IP blocking — rotation through a proxy pool (residential proxies). Minimum 10–15 IPs in pool for catalog of 1000+ items. Request frequency: no more than 1 request per 3–5 seconds per domain.
How Parsing Automation Affects Conversion?
Quick response to competitor price changes helps maintain search positions and retain customers. The parser loads updates into the infoblock, and a Bitrix agent generates a deviation summary >5% for the manager. Reduction in manual data collection costs — up to 80%. Our data scraping respects robots.txt and is compliant with fair use.
What We Collect
Typical data set for competitor parsing:
- Product name and SKU
- Current price (base + discount)
- Availability / quantity
- Link to source product
- Last update date
In Bitrix, this maps to infoblock properties. We recommend adding a COMPETITOR_URL property of type "String" and COMPETITOR_PRICE_DATE of type "Date" — to track data freshness. Competitive analysis is streamlined with automatic price alerts.
Case Study: Electronics Store, 3 Competitors (from Our Practice)
Task: track prices of 2400 SKUs from three competitors, update data every 6 hours. Implementation:
- PHP + Guzzle parser for two static HTML competitors
- Puppeteer for the third (JS-SPA on Vue)
- Cron every 6 hours, sequential startup per competitor with 2-hour pause between them
- Infoblock
COMPETITORS_PRICESlinked to main catalog viaXML_ID - Bitrix agent runs comparison and generates report in HL-block
Result: response time to competitor price change — 6 hours instead of manual monitoring once a week. Manager receives deviation summary >5% via email through \Bitrix\Main\Mail\Event module.
Example parser configuration
Configuration is stored as JSON with competitor URL, type (static or JS), CSS selector, interval hours, and proxy pool. No placeholder URLs used.
How to Avoid Cluttering the Main Catalog with Competitor Data?
Use a separate infoblock COMPETITORS_CATALOG without duplicating elements in b_iblock_element. Link via XML_ID — this is a clean solution without performance loss. We apply this approach in all projects — it ensures main catalog integrity.
What's Included in the Work
| Component | Description |
|---|---|
| Parser for one competitor | Site analysis, technology selection, implementation with proxy rotation |
| Infoblock integration | Create infoblock structure, field mapping, update agent |
| Monitoring and alerts | Cron setup, failure notification, dashboard in Bitrix24 |
| Documentation | Configuration description, instructions for adding competitors |
| Training | Session for managers: how to read reports, configure triggers |
| Stable operation guarantee | 3 months of support and bug fixes when competitor changes layout |
Timeline
| Stage | Duration |
|---|---|
| Competitor site analysis, technology selection | 4–8 hours |
| Parser development (1 competitor, static HTML) | 1–2 days |
| Parser development with headless browser | 2–3 days |
| Integration with Bitrix infoblock | 1 day |
| Cron, monitoring, alerts setup | 4–6 hours |
| Testing on real data | 1 day |
Total for 3 competitors with different technologies — 5–8 business days. Post-launch parser support is mandatory: competitors change layouts. With over 7 years of Bitrix experience and 50+ parsing projects, we deliver reliable solutions. We've been on the market since 2016. Contact us for a consultation and preliminary assessment. Request an analysis — we'll prepare a commercial proposal for your store.