Crawler for Competitor Site Structure: Automated Analysis

You spend days manually collecting 200 competitor URLs, writing down headings and meta descriptions. A month later, a rebranding — and you start all over again. A crawler solves this in minutes and can be reproduced automatically. We are a team with 7 years of experience in web scraping: we have delivered 15+ such solutions for various niches.

One frequent problem is incomplete collection due to JavaScript rendering. Even a static site may contain dynamic elements invisible to a plain HTTP request. A crawler with a headless browser reveals the real structure, including lazy loading. The result is a complete map of the competitor's site: all URLs, headings, meta tags, Schema.org. This approach saves up to 20 hours of work per week and gives an edge in SEO analysis.

What problems does the crawler solve for site structure?

A common pain point — incomplete structure collection due to JavaScript rendering, URL nesting, or canonical duplicates. For example, an ecommerce store on Vue.js may serve identical content on different URLs, distorting the site map. A headless browser crawler detects the real structure, including dynamic loads.

Another issue — Schema.org analysis. Without structured data, it's impossible to assess how the competitor uses rich snippets. The crawler collects JSON-LD and Microdata, allowing you to replicate successful patterns.

What architecture do we use for crawling?

Two working options: Python + Scrapy/Playwright for complex SPAs with lazy loading, Node.js + Puppeteer/Cheerio for most standard sites. For tasks without dynamic JS rendering, an HTTP client with an HTML parser suffices — 5–10 times faster, simpler to deploy.

Minimal Python implementation based on requests + lxml:

import requests
from lxml import html
from urllib.parse import urljoin, urlparse
from collections import deque
import time
from urllib.robotparser import RobotFileParser

class SiteStructureCrawler:
    def __init__(self, base_url: str, max_depth: int = 4, delay: float = 1.0):
        self.base_url = base_url
        self.domain = urlparse(base_url).netloc
        self.max_depth = max_depth
        self.delay = delay
        self.visited: dict[str, dict] = {}
        self.queue: deque = deque([(base_url, 0)])
        # Check robots.txt
        rp = RobotFileParser()
        rp.set_url(f'{base_url}/robots.txt')
        rp.read()
        self.rp = rp

    def crawl(self):
        session = requests.Session()
        session.headers['User-Agent'] = (
            'Mozilla/5.0 (compatible; SiteAnalyzer/1.0; +https://example.com/bot)'
        )
        while self.queue:
            url, depth = self.queue.popleft()
            if url in self.visited or depth > self.max_depth:
                continue
            if not self.rp.can_fetch('*', url):
                continue  # skip disallowed paths
            try:
                resp = session.get(url, timeout=10, allow_redirects=True)
                resp.raise_for_status()
            except requests.RequestException as e:
                self.visited[url] = {'error': str(e), 'depth': depth}
                continue
            doc = html.fromstring(resp.content)
            doc.make_links_absolute(url)
            title = doc.findtext('.//title') or ''
            h1 = [h.text_content().strip() for h in doc.cssselect('h1')]
            meta_desc_el = doc.cssselect('meta[name="description"]')
            meta_desc = meta_desc_el[0].get('content', '') if meta_desc_el else ''
            canonical_el = doc.cssselect('link[rel="canonical"]')
            canonical = canonical_el[0].get('href', '') if canonical_el else ''
            noindex = bool(doc.cssselect('meta[name="robots"][content*="noindex"]'))
            # Collect Schema.org and headings
            schemas = []
            for script in doc.cssselect('script[type="application/ld+json"]'):
                try:
                    import json
                    data = json.loads(script.text_content())
                    schemas.append(data)
                except json.JSONDecodeError:
                    pass
            headings = []
            for tag in ['h1', 'h2', 'h3', 'h4']:
                for el in doc.cssselect(tag):
                    headings.append({'tag': tag, 'text': el.text_content().strip()})
            links = []
            for a in doc.cssselect('a[href]'):
                href = a.get('href', '').strip()
                parsed = urlparse(href)
                if parsed.netloc == self.domain and href not in self.visited:
                    links.append(href)
                    if depth + 1 <= self.max_depth:
                        self.queue.append((href, depth + 1))
            self.visited[url] = {
                'depth': depth,
                'status': resp.status_code,
                'title': title.strip(),
                'h1': h1,
                'meta_description': meta_desc,
                'canonical': canonical,
                'noindex': noindex,
                'internal_links': links,
                'content_type': resp.headers.get('Content-Type', ''),
                'schema': schemas,
                'headings': headings,
            }
            time.sleep(self.delay)
        return self.visited

Working with JavaScript rendering

If the competitor site is an SPA (React/Vue/Angular) or uses lazy-load for main content, a plain HTTP crawler returns empty pages. Here you need a headless browser:

from playwright.sync_api import sync_playwright

def crawl_spa_page(url: str) -> dict:
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto(url, wait_until='networkidle', timeout=30000)
        title = page.title()
        h1_elements = page.query_selector_all('h1')
        h1_texts = [el.inner_text() for el in h1_elements]
        # Collect all links after rendering
        links = page.eval_on_selector_all(
            'a[href]',
            'els => els.map(e => e.href)'
        )
        browser.close()
        return {'title': title, 'h1': h1_texts, 'links': links}

Playwright adds ~2–5 seconds per page vs 0.3–0.8 seconds for plain HTTP. When crawling 500+ pages, this is noticeable — it's used only where necessary.

How to automate regular crawling?

One-time data collection quickly becomes outdated. Competitors change structure, add sections, reformat headings. It is useful to set up automatic runs every week/month and compare results:

def diff_structures(old: dict, new: dict) -> dict:
    added = {url: data for url, data in new.items() if url not in old}
    removed = {url: data for url, data in old.items() if url not in new}
    changed = {}
    for url in old:
        if url in new:
            if old[url].get('title') != new[url].get('title'):
                changed[url] = {
                    'old_title': old[url].get('title'),
                    'new_title': new[url].get('title'),
                }
    return {'added': added, 'removed': removed, 'changed': changed}

Why is Schema.org collection important?

Structured data is a direct indicator of how much a competitor invests in SEO. Having Article, Product, BreadcrumbList, FAQPage gives an advantage in search results. The crawler captures all markup types, allowing you to adopt successful schemas.

Setup and running the crawler

To start collection, follow these steps:

1. Clone the repository with the ready crawler.
2. Install dependencies: pip install requests lxml (or playwright for SPA).
3. Specify the starting URL and maximum crawl depth.
4. Run the script: python crawler.py.
5. After completion, get a report — a JSON or CSV file.

Results and automation

The collected structure can be exported in several formats depending on the task:

import json
import csv

# JSON — for programmatic processing
with open('competitor_structure.json', 'w', encoding='utf-8') as f:
    json.dump(crawler.visited, f, ensure_ascii=False, indent=2)

# CSV — for analysis in Excel/Google Sheets
fieldnames = ['url', 'depth', 'status', 'title', 'meta_description', 'h1', 'noindex', 'canonical']
with open('competitor_structure.csv', 'w', newline='', encoding='utf-8') as f:
    writer = csv.DictWriter(f, fieldnames=fieldnames, extrasaction='ignore')
    writer.writeheader()
    for url, data in crawler.visited.items():
        row = {'url': url, **data}
        if isinstance(row.get('h1'), list):
            row['h1'] = ' | '.join(row['h1'])
        writer.writerow(row)

What is included in the work?

• Development of a crawler tailored to your niche specifics: stack selection (Python or Node.js), depth configuration, delays, robots.txt.
• Integration with a headless browser for SPA sites.
• Collection of all URLs, H1-H6 headings, meta tags, canonical, noindex, Schema.org.
• Export to JSON, CSV, or SQLite as per your choice.
• Automation of runs via cron/Airflow with change history.
• Operational documentation and consultation on result analysis.

Timelines and guarantees

Basic crawler (HTTP, no SPA) with CSV/JSON export — from 1 to 2 business days. With JavaScript rendering support, Schema.org collection, diff comparison, and SQLite storage — from 3 to 4 days. Integration with scheduler and change notifications — additional 1 to 2 days.

We guarantee the crawler respects robots.txt (Robots.txt) — mandatory check before every request. A delay between requests (minimum 1 second) prevents IP blocking. For regular crawling, we rotate User-Agent and proxies.

Typical mistakes in crawler development:

• Ignoring robots.txt — risk of IP blocking.
• Too fast crawling (delay < 1 sec) — server ban.
• Skipping canonical check — duplicates inflate structure.
• No handling of cyclic links — infinite crawl.
• Not accounting for pagination (page/2, page/3) — incomplete collection.

Contact us for a consultation. Order a custom crawler for your niche. Get an analysis of your competitors and crawling recommendations.