Website Crawler for Internal Content Indexing

Website Crawler for Internal Content Indexing

We build custom crawlers that index your website content automatically. Our crawlers traverse all pages, extract metadata, and store results in PostgreSQL, Elasticsearch, or Meilisearch to power site search, content audits, and SEO analysis. Delivery takes three to five working days. We have built crawlers for content portals, e-commerce sites, and knowledge bases. Our clients use the indexed data for real-time search, automated sitemap generation, and regular content quality reports.

Every content-heavy website needs a way to know what it contains. A custom crawler gives you a complete, structured map of your site: all URLs, their titles, descriptions, headings, and text content. This powers faster site search, automated duplicate detection, broken link reports, and hreflang validation.

What's Included in Our Crawler Development Service

We deliver the crawler as a turnkey system. The scope covers:

• Async crawler using Python with asyncio and httpx for high-speed traversal
• HTML parsing with BeautifulSoup to extract title, description, canonical, H1, and body text
• Link graph construction for internal link analysis
• Configurable depth limit, domain restriction, and URL exclusion patterns
• Storage integration: PostgreSQL with tsvector, Elasticsearch, or Meilisearch
• Incremental re-crawl mode that processes only changed pages
• CLI interface for manual runs and scheduling via cron
• Detailed run report: pages found, errors, redirect chains, and indexing statistics

Why Build a Custom Crawler Instead of Using Off-the-Shelf Tools?

General-purpose crawlers like Screaming Frog export CSV files. They do not integrate with your application database. They cannot push new content into your search index in real time. They do not know which content types matter for your specific use case.

A custom crawler reads your site and writes structured data exactly where your application needs it. It runs on your infrastructure, respects your authentication and rate limits, and produces the exact data schema your team designed. Build once, run on schedule, no seat licenses.

Technical Architecture

The crawler is built as an async Python application. It manages a frontier queue of pending URLs, tracks visited URLs to avoid duplicates, and processes pages concurrently with a configurable concurrency limit.

import asyncio import httpx from bs4 import BeautifulSoup from urllib.parse import urljoin, urlparse class SiteCrawler: def __init__(self, base_url: str, concurrency: int = 10): self.base_url = base_url self.domain = urlparse(base_url).netloc self.semaphore = asyncio.Semaphore(concurrency) self.visited: set[str] = set() self.queue: asyncio.Queue = asyncio.Queue() self.results: list[dict] = [] async def fetch_page(self, client: httpx.AsyncClient, url: str) -> dict | None: async with self.semaphore: try: response = await client.get(url, timeout=15, follow_redirects=True) if response.status_code != 200: return {'url': url, 'status': response.status_code, 'error': 'non-200'} return self.parse_html(url, response.text, response.status_code) except Exception as e: return {'url': url, 'status': None, 'error': str(e)} def parse_html(self, url: str, html: str, status: int) -> dict: soup = BeautifulSoup(html, 'html.parser') links = [ urljoin(url, a['href']) for a in soup.find_all('a', href=True) if urlparse(urljoin(url, a['href'])).netloc == self.domain ] return { 'url': url, 'status': status, 'title': (soup.find('title') or {}).get_text(strip=True), 'description': (soup.find('meta', {'name': 'description'}) or {}).get('content', ''), 'h1': (soup.find('h1') or {}).get_text(strip=True), 'canonical': (soup.find('link', {'rel': 'canonical'}) or {}).get('href', ''), 'body_text': soup.get_text(separator=' ', strip=True)[:5000], 'links': links, } 

Saving to Search Index

Results go into the search backend your team prefers. For PostgreSQL full-text search:

CREATE TABLE page_index ( url TEXT PRIMARY KEY, title TEXT, description TEXT, h1 TEXT, body_text TEXT, tsv tsvector GENERATED ALWAYS AS ( to_tsvector('english', coalesce(title, '') || ' ' || coalesce(description, '') || ' ' || coalesce(body_text, '')) ) STORED, crawled_at TIMESTAMPTZ DEFAULT now() ); CREATE INDEX ON page_index USING gin(tsv); 

For Meilisearch, results are pushed via the Python client after each crawl run. Meilisearch handles typo tolerance and faceted filtering without additional configuration.

How Does an Incremental Crawl Work?

Full site crawls can take minutes on large sites. Incremental mode fetches only pages where the content has changed since the last crawl. We detect changes by comparing the ETag or Last-Modified response header, or by checksumming the extracted text. Changed pages are re-indexed; unchanged pages are skipped.

This makes the crawler suitable for running on a schedule every hour without excessive server load.

What Reports Does the Crawler Generate?

After each run the crawler produces a structured report:

1. Total pages crawled and crawl duration
2. HTTP error breakdown: 404 pages, 500 errors, redirect chains longer than two hops
3. Pages missing title, description, or H1
4. Duplicate title and description groups
5. Internal link graph statistics: pages with no inbound links, pages with too many outbound links

Contact us to discuss your indexing requirements. We will review your site structure, choose the right stack, and deliver a working crawler with documentation and deployment instructions.