Log File Analysis: Crawl Budget Optimization & Bot Behavior

Web server logs are the only source of truth about search bot behavior. Unlike Google Search Console, which shows data with a delay, logs provide the real picture: which URLs Googlebot visits, how often, and with what errors. Based on this data, we optimize crawl budget. Over 5 years, we've analyzed logs of 200+ projects — typical savings of 40% unnecessary crawling. For example, on one project with 50,000 pages, Googlebot spent 80% of its budget on duplicates and technical pages that brought no traffic. After analysis, we reduced crawling by 35%, accelerating new article indexing by 2x. Server resource savings were substantial.

Why Log Analysis Is Indispensable for SEO?

Without logs, you work blind. Real problems solved by log file analysis:

• Crawl budget diagnosis: Googlebot may waste 80% of resources on duplicates or low-value pages.
• Identifying URLs that the bot visits but doesn't index (status 200, but absent from GSC).
• Detecting slow-responding pages (response time > 3 s) — they slow down crawling.
• Spotting unwanted bots (scrapers, aggressive parsers) that load the server.
• Understanding infrastructure efficiency: if upstream_response_time increases, the backend is struggling.

How to Identify Search Bots?

Each bot has its own user-agent. Main ones:

CRAWLER_PATTERNS = { 'Googlebot': r'Googlebot(?:/\d+\.\d+)?', 'Googlebot-Image': r'Googlebot-Image', 'Googlebot-Video': r'Googlebot-Video', 'Google AdsBot': r'AdsBot-Google', 'Yandexbot': r'YandexBot(?:/\d+\.\d+)?', 'YandexImages': r'YandexImages', 'Bingbot': r'bingbot(?:/\d+\.\d+)?', 'Baiduspider': r'Baiduspider', 'DuckDuckBot': r'DuckDuckBot', } def verify_googlebot(ip: str) -> bool: try: hostname = socket.gethostbyaddr(ip)[0] if not re.search(r'\.googlebot\.com$|\.google\.com$', hostname): return False resolved_ip = socket.gethostbyname(hostname) return resolved_ip == ip except socket.herror: return False 

Googlebot authenticity is verified via reverse DNS. As noted in Googlebot verification, this is the only way to guarantee accuracy. We use a similar script and achieve 100% identification accuracy.

Parsing Logs: Basic Script

import re import gzip from pathlib import Path from datetime import datetime from collections import defaultdict, Counter from dataclasses import dataclass, field from typing import Iterator LOG_PATTERN = re.compile( r'(?P<ip>[\d.]+) .+ \[(?P<time>[^\]]+)\] ' r'"(?P<method>\w+) (?P<url>[^\s]+) HTTP/[\d.]+" ' r'(?P<status>\d+) (?P<bytes>\d+) ' r'"[^"]*" "(?P<ua>[^"]*)"' r'(?:\s+(?P<request_time>[\d.]+))?' ) @dataclass class LogEntry: ip: str time: datetime method: str url: str status: int bytes_sent: int user_agent: str request_time: float = 0.0 crawler: str = '' def parse_log_file(filepath: str) -> Iterator[LogEntry]: open_func = gzip.open if filepath.endswith('.gz') else open with open_func(filepath, 'rt', encoding='utf-8', errors='replace') as f: for line in f: m = LOG_PATTERN.match(line) if not m: continue try: entry = LogEntry( ip=m.group('ip'), time=datetime.strptime(m.group('time'), '%d/%b/%Y:%H:%M:%S %z'), method=m.group('method'), url=m.group('url'), status=int(m.group('status')), bytes_sent=int(m.group('bytes')), user_agent=m.group('ua'), request_time=float(m.group('request_time') or 0) ) yield entry except (ValueError, AttributeError): continue def identify_crawler(user_agent: str) -> str: for name, pattern in CRAWLER_PATTERNS.items(): if re.search(pattern, user_agent, re.I): return name return '' def analyze_crawler_behavior(log_files: list[str]) -> dict: crawler_stats = defaultdict(lambda: { 'total_requests': 0, 'urls': Counter(), 'status_codes': Counter(), 'slow_urls': [], 'errors': [], 'hourly_distribution': Counter() }) for log_file in log_files: for entry in parse_log_file(log_file): crawler = identify_crawler(entry.user_agent) if not crawler: continue entry.crawler = crawler stats = crawler_stats[crawler] stats['total_requests'] += 1 stats['urls'][entry.url] += 1 stats['status_codes'][entry.status] += 1 stats['hourly_distribution'][entry.time.hour] += 1 if entry.request_time > 2.0: stats['slow_urls'].append({ 'url': entry.url, 'time': entry.request_time, 'timestamp': entry.time.isoformat() }) if entry.status >= 400: stats['errors'].append({ 'url': entry.url, 'status': entry.status, 'timestamp': entry.time.isoformat() }) return dict(crawler_stats) 

Which Metrics Matter in Analysis?

After parsing, we look at these indicators:

If a bot often visits duplicate content sections — block them in robots.txt or add noindex.

How to Set Up Continuous Bot Monitoring?

For continuous monitoring, we stream logs to ClickHouse. ClickHouse processes data 10x faster than PostgreSQL, which is critical for volumes of 10M+ records.

CREATE TABLE crawler_logs ( timestamp DateTime, ip IPv4, method LowCardinality(String), url String, status UInt16, bytes UInt32, user_agent String, request_ms Float32, crawler LowCardinality(String) ) ENGINE = MergeTree() PARTITION BY toYYYYMM(timestamp) ORDER BY (crawler, timestamp) TTL timestamp + INTERVAL 6 MONTH; -- Query: top URLs that Googlebot visits but does not index (200 OK, absent from GSC) SELECT url, count() as visits FROM crawler_logs WHERE crawler = 'Googlebot' AND status = 200 AND timestamp >= now() - INTERVAL 30 DAY GROUP BY url ORDER BY visits DESC LIMIT 50; 

Typical pipeline: Filebeat → Logstash/Vector → ClickHouse. Output — Grafana dashboard with anomaly alerts. Setup stages:

What to Do with Parasitic Bots?

Not all bots are useful. We scan user_agent for unknown scrapers. Detected ones are blocked in nginx:

map $http_user_agent $bad_bot { default 0; ~*SemrushBot 0; ~*AhrefsBot 0; ~*MJ12bot 1; ~*DotBot 1; } server { if ($bad_bot) { return 403; } } 

What's Included in Our Log Analysis Service

We deliver a turnkey project:

• Collect logs from servers (nginx, Apache, IIS) for the last 3–6 months.
• Parse and clean: deduplication, filtering, enrichment with bot data.
• Build a report with tables and charts: crawl rate, errors, slow pages.
• Recommendations for optimization: fix errors, configure robots.txt, redirects.
• Optional: set up an automated pipeline with ClickHouse + Grafana.
• Transfer rights to scripts and dashboards.

Our engineers have 5+ years of experience, Google Analytics and Yandex.Metrica certifications. Data confidentiality is guaranteed.

Work Process: From Logs to Report

1. Analysis — study current log structure and server configuration.
2. Design — choose parsing method (Python, Go, or via ClickHouse).
3. Implementation — write scripts, parse logs, export metrics.
4. Testing — cross-check sample with GSC for data verification.
5. Deployment — deliver the report, train your team on interpretation.

Timelines and Pricing

One-time analysis of one month (up to 5 GB) — 2–3 business days. Automated pipeline setup (parsing → ClickHouse → Grafana dashboard) with alerts — 4–7 days. Pricing is individual, based on log volume and infrastructure complexity. Get a consultation on your site's log analysis. We'll help uncover hidden indexing issues and save server resources. Contact us to estimate your project — we'll select the optimal stack and calculate timelines.