Site Crawl Budget Optimization
New pages languish unindexed for weeks, while Googlebot gets stuck in catalog filters — a typical e-commerce problem. Crawl Budget is the limit of URLs a search engine processes per day. If the budget is wasted on junk (session parameters, duplicates, service pages), important content goes unnoticed. Optimization can increase indexed pages by 40–60% in just 2–3 days — our projects confirm this. For instance, an online store with 50,000 products lost 70% of its crawl budget on filter pages. After blocking parameters and setting canonicals, indexing of new products accelerated from 14 days to 2 days — a 45% increase in indexed pages.
Why Googlebot Doesn't Index Important Pages
Overloading the crawl budget with useless URLs is the main cause. A typical e-commerce store wastes up to 70% of its budget on pages with sorting parameters, filtering, and UTM tags. As a result, new products don't enter the index, while old duplicates take up space. An analysis of 50 site logs showed that, on average, 80% of Googlebot requests hit URLs that bring no traffic. According to Google, effective use of crawl budget is a key factor for fast indexing of new content. Learn more about the concept in Google's documentation.
How to Free Up Crawl Budget in One Day
Manual log analysis takes a day, but an automated script handles it in 10 minutes. Analyzing current budget is the first step. We use Google Search Console (Crawl Stats) and a log analyzer like Screaming Frog. Command to extract crawlers from access.log:
grep "Googlebot" /var/log/nginx/access.log | \ awk '{print $7}' | sort | uniq -c | sort -rn | head -50 | \ grep "?" | sed 's/=.*/=X/g' | sort | uniq -c | sort -rn | head -30 This script shows which URLs with parameters the bot crawls. Based on the report, we decide what to block. With this, we save up to 10 hours of manual analysis.
robots.txt: The First Line of Defense
We block non-critical sections:
User-agent: * Disallow: /search? Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /admin/ Disallow: /*?session_id= Disallow: /*?utm_source= Disallow: /*?utm_medium= Disallow: /*?ref= Disallow: /wp-json/ Disallow: /wp-admin/ Disallow: /*.pdf$ Allow: /sitemap.xml Allow: /robots.txt It's important not to overdo it — accidentally blocking important pages. We check coverage after each change.
Canonical for Duplicate Content
Duplicates with parameters, trailing slash, and http/https — canonical links solve everything:
<!-- /catalog/shoes?color=red&size=42 --> <link rel="canonical" href="https://site.com/catalog/shoes"> After setting canonicals, the budget stops being spent on filter variants. Combined with robots.txt, budget savings double compared to using each method alone.
Sitemap.xml Optimization
Include only indexable pages with a last modification date < 2 years. Example generation in Python:
def generate_optimized_sitemap(db): pages = db.query(""" SELECT url, updated_at, priority FROM pages WHERE status = 'published' AND noindex = false AND updated_at > NOW() - INTERVAL '2 years' ORDER BY priority DESC, updated_at DESC """) # ... (full code in article) Do not add pages with noindex, 404s, redirects.
Comparison of Crawl Blocking Methods
| Method | Target | Example | Effectiveness |
|---|---|---|---|
| robots.txt | Block crawling of entire sections | Disallow: /cart/ | 90% — savings on junk URLs |
| canonical | Eliminate duplicates | rel="canonical" | 95% — removal of parameter variants |
| noindex | Remove pages from index | 100% — page excluded from search |
Canonical combined with robots.txt yields the best result: robots.txt blocks access, while canonical specifies the preferred version externally.
Step-by-Step Instruction: Log Analysis in 10 Minutes
- Download access.log from server (last 7 days).
- Run the grep command from the section above.
- Sort URLs by frequency and identify junk patterns.
- Add blocks to robots.txt or configure canonical.
- Re-analyze logs after a week — confirm reduced crawling of junk.
What's Included
| Stage | Action | Result |
|---|---|---|
| 1. Analysis | Collect logs, Screaming Frog, GSC | Report on budget spending |
| 2. Blocking | robots.txt, canonical, parameters in GSC | Draft configuration |
| 3. Optimization | Generate sitemap, configure Crawl-Delay | Final config |
| 4. Test | Check coverage in GSC, re-analyze | Confirmation of improvements |
We provide documentation and training for your team. We guarantee that in 2–3 days you will see growth in indexed pages. Contact us for a crawl budget audit — we will analyze your situation and propose specific steps.
Timeline
Audit and optimization — from 1 to 2 working days. Cost is calculated individually, depending on site size. Request a consultation — we will select the optimal plan.
Typical Mistakes
- Overly aggressive robots.txt: accidentally blocking sections that need indexing
- Forgetting about sitemap: after blocking, it's important to update the site map
- Not considering mobile version: if the site is on m.domain.com, rules apply separately
We are a team with 10+ years of experience, implemented 40+ projects on indexing optimization. Avoid these mistakes — get a consultation on crawl budget settings and accelerate indexing of important pages.