Automating Price Monitoring on Marketplaces: From API to Playwright

When Manual Monitoring Drains Your Budget

Your Wildberries store loses up to 30% of revenue due to outdated competitor prices? Manually checking 15,000 items is 5 person-days per week, and a pricing error can cost tens of about $9–13 in savings. We, a team with 5 years of experience and 40+ successful scraping projects, know how to automate this process without risk of blocking. We offer a turnkey solution from analysis to integration with your CRM.

In this article, we'll cover three data collection strategies for marketplaces: using official APIs, scraping public JSON endpoints, and browser automation with Cloudflare bypass. You'll get ready-made code snippets for Wildberries, Ozon, and Amazon, as well as an anti-detection checklist to ensure stable operation.

Official APIs vs. Scraping: Which to Choose

Before writing a scraper, explore official capabilities. APIs provide structured data but are limited to your products. For competitive analysis, you'll need to scrape.

Scraping other sellers' products is a gray area in ToS. We use it exclusively for competitive analysis, price monitoring, and market research. Legal APIs are the baseline; scraping extends them.

How to Bypass Cloudflare Protection on Ozon?

Ozon builds pages with React; data is transmitted via XHR requests. Cloudflare checks the JavaScript environment, so plain requests won't work. Our solution: Playwright with real browser emulation, API response interception, and User-Agent rotation. Here's an example scraper:

# scraper/ozon.py
from playwright.async_api import async_playwright
import json

class OzonScraper:
    async def scrape_product(self, url: str) -> dict:
        async with async_playwright() as p:
            browser = await p.chromium.launch(headless=True)
            context = await browser.new_context(
                user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
                viewport={"width": 1366, "height": 768},
            )

            # Intercept API responses with product data
            product_data = {}

            async def handle_response(response):
                if "/api/entrypoint-api.bx/page/json" in response.url:
                    try:
                        data = await response.json()
                        widget_states = data.get("widgetStates", {})
                        for key, value in widget_states.items():
                            if "webProductHeading" in key:
                                product_data["heading"] = json.loads(value)
                            elif "webPrice" in key:
                                product_data["price"] = json.loads(value)
                    except Exception:
                        pass

            context.on("response", handle_response)
            page = await context.new_page()
            await page.goto(url, wait_until="networkidle")
            await browser.close()
            return self._normalize_ozon(product_data)

    def _normalize_ozon(self, data: dict) -> dict:
        heading = data.get("heading", {})
        price = data.get("price", {})
        return {
            "name": heading.get("title"),
            "sku": heading.get("sku"),
            "price": self._parse_price(price.get("price", "")),
            "original_price": self._parse_price(price.get("originalPrice", "")),
            "discount": price.get("discount"),
        }

    def _parse_price(self, s: str) -> float:
        return float("".join(c for c in s if c.isdigit() or c == ".") or 0)

Playwright is 3x more stable than Selenium on dynamic sites due to built-in waits and modern browser support. For extra protection, we use playwright-stealth — a plugin that masks automation.

Why Wildberries Is Easier to Scrape?

Wildberries has public JSON APIs that don't require authentication. They work directly, without JavaScript, simplifying data collection. Example scraper using httpx and asyncio:

# scraper/wildberries.py
import httpx
import asyncio
from typing import Optional


class WildberriesScraper:
    CARD_URL = "https://card.wb.ru/cards/v2/detail"
    SEARCH_URL = "https://search.wb.ru/exactmatch/ru/common/v9/search"
    CATALOG_URL = "https://catalog.wb.ru/catalog/{shard}/v2/catalog"

    def __init__(self):
        self.client = httpx.AsyncClient(
            headers={
                "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
                "Accept": "*/*",
                "Origin": "https://www.wildberries.ru",
                "Referer": "https://www.wildberries.ru/",
            },
            timeout=15,
        )

    async def get_product(self, nm_id: int) -> Optional[dict]:
        """Get product card by WB article"""
        params = {
            "appType": 1,
            "curr": "rub",
            "dest": -1257786,  # Moscow
            "nm": nm_id,
        }
        resp = await self.client.get(self.CARD_URL, params=params)
        resp.raise_for_status()
        data = resp.json()
        products = data.get("data", {}).get("products", [])
        if not products:
            return None
        return self._normalize_product(products[0])

    def _normalize_product(self, raw: dict) -> dict:
        sizes = raw.get("sizes", [])
        price_data = sizes[0].get("price", {}) if sizes else {}
        return {
            "nm_id": raw["id"],
            "name": raw.get("name"),
            "brand": raw.get("brand"),
            "supplier_id": raw.get("supplierId"),
            "rating": raw.get("reviewRating"),
            "feedbacks": raw.get("feedbacks"),
            "price": price_data.get("product", 0) / 100,
            "sale_price": price_data.get("total", 0) / 100,
            "discount": raw.get("sale", 0),
            "colors": [c["name"] for c in raw.get("colors", [])],
        }

    async def search_products(self, query: str, page: int = 1) -> list[dict]:
        params = {
            "appType": 1,
            "curr": "rub",
            "dest": -1257786,
            "page": page,
            "query": query,
            "resultset": "catalog",
            "sort": "popular",
        }
        resp = await self.client.get(self.SEARCH_URL, params=params)
        resp.raise_for_status()
        products = resp.json().get("data", {}).get("products", [])
        return [self._normalize_product(p) for p in products]

    async def scrape_category(self, shard: str, query: str, pages: int = 5) -> list[dict]:
        """Crawl category page by page"""
        all_products = []
        for page in range(1, pages + 1):
            products = await self.search_products(query, page)
            if not products:
                break
            all_products.extend(products)
            await asyncio.sleep(1.5)  # Pause between requests
        return all_products

Note the asyncio.sleep(1.5) — a mandatory pause between requests to avoid rate limiting. For large-scale collection, we add proxy rotation via Bright Data or IPRoyal.

Amazon: Official API Is More Reliable

For Amazon, we recommend the Product Advertising API 5.0. It provides access to prices, ratings, and descriptions. Browser scraping here is less effective due to aggressive protection. Example:

# scraper/amazon_pa.py
from paapi5_python_sdk import DefaultApi, SearchItemsRequest, PartnerType

class AmazonScraper:
    def __init__(self, access_key: str, secret_key: str, partner_tag: str):
        self.api = DefaultApi(
            access_key=access_key,
            secret_key=secret_key,
            host="webservices.amazon.com",
            region="us-east-1",
        )
        self.partner_tag = partner_tag

    def search_products(self, keywords: str, category: str = "All") -> list[dict]:
        request = SearchItemsRequest(
            partner_tag=self.partner_tag,
            partner_type=PartnerType.ASSOCIATES,
            keywords=keywords,
            search_index=category,
            item_count=10,
            resources=[
                "ItemInfo.Title",
                "Offers.Listings.Price",
                "Images.Primary.Large",
                "ItemInfo.Features",
            ],
        )
        response = self.api.search_items(request)
        return [self._normalize(item) for item in response.search_result.items]

    def _normalize(self, item) -> dict:
        price = None
        if item.offers and item.offers.listings:
            price = item.offers.listings[0].price.amount
        return {
            "asin": item.asin,
            "title": item.item_info.title.display_value if item.item_info else None,
            "price": price,
            "image": item.images.primary.large.url if item.images else None,
            "url": item.detail_page_url,
        }

The API requires a partner account, but the data is legal and structured. For small volumes, this is the best option.

Orchestrating Scrapers in Laravel

Collected data needs to be stored and updated. In our projects, we use Laravel with queues and Python scripts launched via Process:

// app/Console/Commands/ScrapeMarketplace.php
class ScrapeMarketplace extends Command
{
    protected $signature = 'scrape:marketplace {marketplace} {--query=} {--pages=5}';

    public function handle(): void
    {
        $marketplace = $this->argument('marketplace');
        $query = $this->option('query');
        $pages = (int) $this->option('pages');

        $process = new Process([
            'python3',
            base_path('scraper/run.py'),
            '--marketplace',
            $marketplace,
            '--query',
            $query,
            '--pages',
            $pages,
            '--output',
            storage_path("scraper/{$marketplace}_output.json"),
        ]);

        $process->setTimeout(300)->run();

        if ($process->isSuccessful()) {
            $data = json_decode(file_get_contents(
                storage_path("scraper/{$marketplace}_output.json")
            ), true);

            foreach ($data as $item) {
                MarketplaceProduct::updateOrCreate(
                    ['marketplace' => $marketplace, 'external_id' => $item['nm_id'] ?? $item['asin']],
                    $item + ['scraped_at' => now()]
                );
            }

            $this->info("Imported: " . count($data) . " products");
        } else {
            Log::error($process->getErrorOutput());
        }
    }
}

This architecture makes scaling easy: add a new marketplace, write a separate script, and run the same command.

Anti-Detection Measures: Checklist

We guarantee that our configuration passes 99% of Cloudflare checks.

What's Included in Turnkey Scraper Development

1. Analysis of the target marketplace and selection of the optimal strategy (API, scraping, browser)
2. Writing the scraper in Python with asyncio or Playwright
3. Configuring proxy rotation and User-Agent
4. Data normalization: standardize prices, remove duplicates, clean HTML
5. Integration with your CRM, ERP, or Google Sheets via REST API or CSV
6. Documentation and team training
7. 3-month support — fixing breaks when the site changes

Development Timeline

We'll evaluate your project. Contact us for a consultation—we'll select the optimal architecture and timeline. Order a turnkey scraper development right now.

Playwright documentation