Screaming Frog Site Crawl Setup
Screaming Frog SEO Spider is desktop crawler, de-facto standard for technical SEO audit. Can crawl hundreds of thousands of URLs, checks redirects, duplicates, broken links, meta tags, heading structure, hreflang, canonical, sitemap, robots.txt.
Installation and License
Screaming Frog distributed as Java application. Free version limited to 500 URLs. Real projects need paid license (~$259/year).
Running via CLI
Key for automation is headless mode:
screamingfrogseospider \
--crawl https://example.com \
--headless \
--save-crawl \
--export-tabs "Internal:All,Response Codes:All" \
--output-folder /var/reports/
Configuration via Config File
Save standardized config and pass on launch for repeatability.
Authentication for Closed Sections
Use cookie authentication or HTTP Basic Auth for protected areas.
Processing Results
Screaming Frog exports separate CSV per tab. Process with pandas for analysis and reporting.
Automatic Weekly Audit
Setup cron job for weekly crawls with automatic result processing and alerts.
Resource Usage
RAM-heavy application. Without JS: 1–2 GB. With rendering: 4–8 GB. Set JVM heap explicitly.
Crawl speed: without JS 5–20 pages/sec, with JS 1–3 pages/sec.
Timeline
Setup regular CLI crawling with result processing and alerts — 2–3 working days.