How to Rotate Proxies in Python Requests: The Complete 2026 Guide

The Two Ways to Rotate Proxies in Python

When developers begin web scraping, they often try client-side rotation: downloading an array of 500 static proxy IPs and using random.choice(proxy_list) on each request. This method quickly fails at scale because:

• 30% to 50% of cheap static proxies are dead at any given second.
• Re-establishing new TCP connections to separate IP hosts adds 800ms+ latency per request.
• Handling proxy failures requires complex custom retry loops.

In contrast, enterprise web scrapers use Backconnect Gateways (like 2Captcha Proxy). Your script connects to one persistent entry server, and the provider's server-side load balancer rotates the egress residential IP on every HTTP request.

1. Basic Python Requests with a Rotating Proxy Gateway

Configuring Python requests with 2Captcha Proxy requires passing a single proxy dictionary to requests.get():

import requests

# 2Captcha Proxy Gateway Credentials
USERNAME = "customer_YOUR_USER_ID"
PASSWORD = "YOUR_PASSWORD"
GATEWAY = "proxy.2captcha.com:8080"

proxies = {
    "http": f"http://{USERNAME}:{PASSWORD}@{GATEWAY}",
    "https": f"http://{USERNAME}:{PASSWORD}@{GATEWAY}"
}

# Each request automatically receives a fresh residential IP!
for i in range(5):
    response = requests.get("https://api.ipify.org?format=json", proxies=proxies)
    print(f"Request {i+1} IP: {response.json()['ip']}")

2. Production Pattern: Persistent Session with Automatic Retries

In production scraping pipelines, never call requests.get() directly in a loop. Always use a requests.Session() combined with urllib3.util.Retry to handle transient connection drops and HTTP 429 (Too Many Requests) / 503 errors gracefully:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

def get_resilient_session(proxy_url: str) -> requests.Session:
    session = requests.Session()
    session.proxies = {
        "http": proxy_url,
        "https": proxy_url
    }
    
    # Configure exponential backoff retry on common rate limits
    retries = Retry(
        total=5,
        backoff_factor=1,
        status_forcelist=[429, 500, 502, 503, 504],
        raise_on_status=False
    )
    
    adapter = HTTPAdapter(max_retries=retries, pool_connections=20, pool_maxsize=50)
    session.mount("http://", adapter)
    session.mount("https://", adapter)
    return session

# Initialize scraper session
proxy_endpoint = f"http://{USERNAME}:{PASSWORD}@proxy.2captcha.com:8080"
scraper = get_resilient_session(proxy_endpoint)

response = scraper.get("https://target-ecommerce.com/products")
print(f"Status: {response.status_code}")

3. Sticky Sessions vs Rotating Sessions

Depending on your workflow, you may need either per-request rotation (for catalog scraping) or sticky sessions (for logging into user accounts or multi-step checkout forms).

• Per-Request Rotation: Every query changes IP. Ideal for scraping search results and product listings.
• Sticky Session (10–120 min): Appending a session ID to your username (e.g. username_session-abc123_time-30) locks your outgoing requests to the same residential IP for 30 minutes.

Summary Checklist for High-Speed Scraping

1. Always set connection timeouts: Pass timeout=(5, 15) to avoid hung requests on slow peer nodes.
2. Reuse TCP connections with `requests.Session()`: Eliminates SSL handshake overhead.
3. Monitor bandwidth usage: Filter images, fonts, and media assets unless strictly necessary to keep residential GB costs low.