Analyzing and Optimizing Page Indexation (Index Coverage)
Index Coverage is the ratio between pages on your site and pages Google actually indexed. Many SEO traffic problems are explained not by content quality, but by pages simply not making it into the index.
Google Search Console: Coverage Report
GSC → Index → Pages divides all discovered URLs into categories:
Indexed — pages in the index (goal: 80–95% of total URLs).
Crawled - currently not indexed — Google saw it but didn't index. Reasons: thin content, low quality, too much similar content.
Discovered - currently not indexed — Google found URL but hasn't crawled yet. Usually due to crawl budget overload.
Not indexed — noindex — explicit no-index directive.
Duplicate — duplicate pages (Google chose canonical).
Excluded by robots.txt — blocked in robots.txt.
Export All URLs and Their Statuses
# Google Search Console API
from googleapiclient.discovery import build
from oauth2client.service_account import ServiceAccountCredentials
credentials = ServiceAccountCredentials.from_json_keyfile_name(
'gsc-credentials.json',
['https://www.googleapis.com/auth/webmasters.readonly']
)
service = build('searchconsole', 'v1', credentials=credentials)
# URL Inspection API — check specific URL
response = service.urlInspection().index().inspect(
body={
'inspectionUrl': 'https://company.com/products/iphone-15',
'siteUrl': 'https://company.com/'
}
).execute()
indexing_state = response['inspectionResult']['indexStatusResult']['indexingState']
# INDEXING_ALLOWED, NOT_INDEXED, etc.
Diagnosing "Crawled - currently not indexed"
Most difficult category to fix — Google sees the page but doesn't deem it indexable.
def diagnose_not_indexed_pages(urls):
issues = []
for url in urls:
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
# Word count
text = ' '.join(soup.get_text().split())
word_count = len(text.split())
# Check title uniqueness
title = soup.find('title').text if soup.find('title') else ''
issues.append({
'url': url,
'word_count': word_count,
'title': title,
'has_canonical': bool(soup.find('link', rel='canonical')),
'robots': soup.find('meta', attrs={'name': 'robots'}),
'h1_count': len(soup.find_all('h1')),
})
# Thin content
thin = [p for p in issues if p['word_count'] < 300]
print(f"Thin content ({len(thin)} pages):")
for p in thin[:10]:
print(f" {p['url']}: {p['word_count']} words")
Fixing Common Issues
Thin content — less than 300 words. Solution: expand or merge similar pages.
Duplicate content without canonical: add canonical to all URL variants.
Noindex by mistake: search for noindex in templates and remove from important pages.
Pages behind authentication: public content must be accessible without cookies.
Sitemap as Indexation Tool
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://company.com/products/iphone-15</loc>
<lastmod>2024-03-15</lastmod>
<changefreq>weekly</changefreq>
<priority>0.9</priority>
</url>
</urlset>
Ping Google about sitemap update:
curl "https://www.google.com/ping?sitemap=https://company.com/sitemap.xml"
IndexNow API (Bing, Yandex):
curl -X POST https://api.indexnow.org/IndexNow \
-H "Content-Type: application/json" \
-d '{
"host": "company.com",
"key": "your-key",
"urlList": ["https://company.com/new-page"]
}'
Monitoring Indexation Dynamics
def track_indexation_rate(history_db):
"""Track % of indexed pages over time"""
total_published = count_published_pages()
indexed_count = get_gsc_indexed_count()
rate = indexed_count / total_published * 100
history_db.save({'date': datetime.now(), 'rate': rate})
# Alert if declining
last_week = history_db.get_7days_ago()
if rate < last_week.rate - 5:
send_alert(f"Index coverage dropped from {last_week.rate:.1f}% to {rate:.1f}%")
Timeline
Index Coverage audit + action plan — 2–3 business days.