Bypassing Anti-Scraping Protections (CAPTCHA, Rate Limiting)

Bypassing Anti-Scraping Protections (CAPTCHA, Rate Limiting)

Industrial anti-scraping protections — DataDome, Cloudflare Bot Management, PerimeterX, Akamai Bot Manager — analyze user behavior across dozens of signals: from WebGL rendering deviations to micro-timings between clicks. Each system uses its own ML models that are updated weekly. Recently, we solved a problem for an e-commerce client: they needed to scrape a competitor's catalog protected by DataDome and reCAPTCHA v3. We applied rotating residential proxies, a modified Playwright with 30+ signature masking, and a custom CAPTCHA solver — stability was 97% at 500,000 pages per day. Such results require deep understanding of the specific protection and combination of several techniques. We implement turnkey bypass: from basic stealth to a full system with monitoring in 12–18 days. Contact us to evaluate your project.

In this article, we'll break down the main types of protections, methods for bypassing rate limiting and CAPTCHA, setting up proxy infrastructure, and ways to maintain stability when algorithms change. Such results require constant monitoring and adaptation — we include this in support.

Classification of Anti-Scraping Protections and How to Tackle Each

Level 1 — Rate limiting. Simple IP-based protection: more than N requests per second → block. Solved by proxy rotation and reducing request frequency.

Level 2 — Browser fingerprinting. Checks navigator.webdriver, canvas fingerprint, WebGL rendering, audio context, plugin list. Detects headless browsers without masking.

Level 3 — Behavioral analysis. ML models on the protection side: mouse movement patterns, timings between actions, event order. Differentiates bots from humans even with correct fingerprint.

Level 4 — CAPTCHA. Visual or behavioral tasks. Google reCAPTCHA v2/v3, hCaptcha, Arkose Labs (FunCaptcha), Cloudflare Turnstile.

We have worked with each of these protections on dozens of projects. Our experience — 5+ years on the market and 30+ successful integrations. We guarantee scraping stability even when protection algorithms are updated. In practice, a combination of several levels is most common, for example, Cloudflare Bot Management + reCAPTCHA v3.

Bypassing Rate Limiting

import asyncio
import random
from aiohttp import ClientSession

async def fetch_with_delay(session, url, semaphore):
    async with semaphore:
        await asyncio.sleep(2 + random.gauss(1, 0.5))  # normal distribution
        async with session.get(url) as resp:
            return await resp.text()

semaphore = asyncio.Semaphore(3)  # max 3 concurrent requests

Random delays with a normal distribution are significantly more effective than fixed ones: the pattern is closer to human behavior.

Stealth Playwright

const { chromium } = require('playwright');
const { stealth } = require('playwright-stealth');
const browser = await chromium.launch({
  args: [
    '--disable-blink-features=AutomationControlled',
    '--no-sandbox',
  ]
});
const context = await browser.newContext({
  userAgent: getRandomUserAgent(),
  locale: 'ru-RU',
  timezoneId: 'Europe/Moscow',
  geolocation: {
    longitude: 37.6173,
    latitude: 55.7558
  },
  permissions: ['geolocation'],
});
await stealth(context);

playwright-stealth patches over 30 detectable fields: navigator.webdriver, window.chrome, navigator.languages, canvas noise, and more. Using stealth mode is mandatory for sites with behavioral analysis.

Why Proxy Quality Is Critical for Scraping

Protections analyze IP cleanliness and behavior. Residential proxies (Bright Data, Oxylabs) — real IPs from home users — are rarely blocked. Mobile proxies (4G/5G) have a high trust score. Datacenter IPs (AWS, DigitalOcean) are often blacklisted and unsuitable for complex protections. For large projects, residential IPs are worth the investment — the cost of a block is high.

class ProxyRotator:
    def __init__(self, proxies: list):
        self.proxies = proxies
        self.stats = {p: {'success': 0, 'fail': 0} for p in proxies}

    def get_best_proxy(self):
        # select proxy with highest success rate
        return max(
            self.proxies,
            key=lambda p: self.stats[p]['success'] / max(self.stats[p]['success'] + self.stats[p]['fail'], 1)
        )

    def report_success(self, proxy):
        self.stats[proxy]['success'] += 1

    def report_fail(self, proxy):
        self.stats[proxy]['fail'] += 1

How to Bypass CAPTCHA Without Risk of Blocking

Automatic solving via services: 2captcha, Anti-Captcha, CapSolver, NopeCHA. For CAPTCHA v2/v3, tokens from services are used. For reCAPTCHA v3, a high score is needed, achieved through a quality browser profile. (Google reCAPTCHA documentation)

from twocaptcha import TwoCaptcha
solver = TwoCaptcha(API_KEY)

# reCAPTCHA v2
result = solver.recaptcha(
    sitekey='6LfXXXXXXXXXXXXXXXXXXXXX',
    url='https://example.com/page'
)
token = result['code']

# insert into form

Working with Cookies and Sessions

Session cookies are an important signal for protections. A bot that doesn't accumulate cookies over multiple pages looks suspicious.

# Save and restore Playwright context
await context.storage_state(path='session.json')
# In next run
context = await browser.new_context(storage_state='session.json')

For complex sites: first "warm up" the session — visit the homepage, a couple of random pages, simulate scrolling — then proceed to target URLs.

Detecting Protection Algorithm Changes

Protections update algorithms. Monitoring is needed:

• Track HTTP statuses: growth of 403/429/503 → trigger check
• Compare fingerprint requests (JavaScript loaded by DataDome)
• Alerts when successful parse rate drops below threshold

What Our Work Includes

• Analysis of the target site and identification of protection type
• Development and configuration of bypass (stealth, proxies, CAPTCHA)
• Integration with your parser (API, SDK)
• Stability monitoring and automatic adaptation
• Documentation and training for your team

Timelines

Basic bypass rate limiting + stealth: 3–5 days. Full system with CAPTCHA solver, proxy rotator, and monitoring: 12–18 days.

Contact us for a consultation — we'll evaluate your project and offer the optimal solution.