Proxy Rotation for Web Scraping: Types, Setup and Best Practices

Why Proxies Matter for Web Scraping

When scraping websites, sending many requests from a single IP address triggers rate limits and blocks. Proxies route your requests through different IP addresses, making your traffic appear to come from multiple sources:

Proxy Types

By source

By rotation model

By protocol

Proxy Rotation Strategies

Basic rotation

Advanced rotation

Python implementation: Basic proxy rotation

import requests
import random
import time
from itertools import cycle

class ProxyRotator:
    """Manage proxy rotation for web scraping."""

    def __init__(self, proxies):
        self.proxies = proxies
        self.working_proxies = list(proxies)
        self.failed_proxies = {}
        self.cycle = cycle(self.working_proxies)

    def get_next_proxy(self, strategy='round_robin'):
        """Return the next proxy based on the selected strategy."""
        if strategy == 'round_robin':
            return next(self.cycle)
        elif strategy == 'random':
            return random.choice(self.working_proxies)
        else:
            return next(self.cycle)

    def mark_failed(self, proxy, cooldown=300):
        """Remove a proxy from rotation temporarily."""
        if proxy in self.working_proxies:
            self.working_proxies.remove(proxy)
            self.failed_proxies[proxy] = time.time() + cooldown
            # Rebuild cycle without the failed proxy
            self.cycle = cycle(self.working_proxies)
            print(f"Proxy removed: {proxy} "
                  f"(cooldown: {cooldown}s, "
                  f"remaining: {len(self.working_proxies)})")

    def restore_proxies(self):
        """Restore proxies whose cooldown has expired."""
        now = time.time()
        restored = []
        for proxy, expiry in list(self.failed_proxies.items()):
            if now >= expiry:
                self.working_proxies.append(proxy)
                restored.append(proxy)
                del self.failed_proxies[proxy]
        if restored:
            self.cycle = cycle(self.working_proxies)
            print(f"Restored {len(restored)} proxies. "
                  f"Pool size: {len(self.working_proxies)}")

    def fetch(self, url, max_retries=3, delay=1):
        """Fetch a URL using proxy rotation with retries."""
        self.restore_proxies()

        for attempt in range(max_retries):
            if not self.working_proxies:
                print("No working proxies available.")
                return None

            proxy = self.get_next_proxy()
            proxy_dict = {
                'http': f'http://{proxy}',
                'https': f'http://{proxy}'
            }

            try:
                response = requests.get(
                    url,
                    proxies=proxy_dict,
                    timeout=15,
                    headers={
                        'User-Agent': (
                            'Mozilla/5.0 (Windows NT 10.0; Win64; x64) '
                            'AppleWebKit/537.36')
                    }
                )
                if response.status_code == 200:
                    return response
                elif response.status_code in (403, 429, 503):
                    self.mark_failed(proxy)
                    time.sleep(delay)
                else:
                    return response
            except requests.RequestException:
                self.mark_failed(proxy)
                time.sleep(delay)

        return None


# Usage
proxies = [
    'user:pass@proxy1.example.com:8080',
    'user:pass@proxy2.example.com:8080',
    'user:pass@proxy3.example.com:8080',
]

rotator = ProxyRotator(proxies)
response = rotator.fetch('https://example.com/page')

Proxy Provider Comparison

Major proxy providers

Cost comparison

Choosing the right proxy type

Anti-Detection Best Practices

Proxies alone do not prevent detection. Combine proxies with other techniques:

Common mistakes that reveal scrapers

Proxy Infrastructure Architecture

Small scale (under 10K requests/day)

Medium scale (10K-1M requests/day)

Large scale (1M+ requests/day)

Legal and Ethical Considerations

Processing Scraped Data

After collecting data through proxied scraping, the raw HTML or downloaded files often contain email addresses and contact information mixed with other content. Upload scraped files to Email Extractor to extract email addresses from HTML, JSON, XML, CSV or text files. The tool handles multiple file formats, so you can process scraped output regardless of the format your scraping pipeline saves it in.

Related Guides

• Rate Limiting and Polite Scraping
• CAPTCHA and Bot Detection in Scraping
• Headless Browser Scraping Comparison
• Email Scraper Python Tutorial
• Building a Scraping Pipeline