Scrapy proxy layer: why a stock spider dies on the third page
Scrapy is brutally efficient. Out of the box it fires sixteen concurrent requests from a single IP address, keeps connections alive, and follows links faster than any human ever could. On an unprotected site that is a feature. On Amazon, Booking.com, Google SERP or LinkedIn it is a signature — and the first thing that breaks is not your parsing logic but your IP reputation.
The typical progression looks the same for everyone: 200 responses, then 429, then 403, then an empty body with a challenge page. Adding a random User-Agent does not help, because the block happened one layer below the browser fingerprint — at the network identity level. That is where a proper scrapy proxy configuration comes in, and where the choice of IP source matters more than the code you write around it.
How rotation actually works inside Scrapy
Scrapy has no built-in pool manager. It ships with HttpProxyMiddleware, which does exactly one thing: it reads the proxy address from request.meta['proxy'] and routes the connection through it. Everything else — pool storage, rotation policy, ban detection, retries — is your responsibility. There are three practical patterns.
1. Per-request assignment through meta
The simplest method: in your spider you import the random module, pick an endpoint from a list, and set request.meta['proxy'] for every yielded Request. Fine for a one-off script of fifty pages. It falls apart the moment you need retry logic, because a failed request retried by Scrapy keeps the same meta and therefore the same dead endpoint.
2. A custom downloader_middleware class
The production approach. You write a small class with a process_request method that injects the proxy, plus process_response and process_exception to catch bans and re-route. Register it in the DOWNLOADER_MIDDLEWARES setting in your settings file, with a priority number below 750 so it runs before HttpProxyMiddleware. Twenty lines of code, full control, no external dependency.
3. The scrapy-rotating-proxies package
The community package does the bookkeeping for you: it tracks which endpoints are alive, backs off from dead ones, and re-checks them later. You add ROTATING_PROXY_LIST (or ROTATING_PROXY_LIST_PATH pointing to a text file), enable RotatingProxyMiddleware and BanDetectionMiddleware, and override the ban detection policy — the default treats only 403 and 429 as bans, while real targets often return 200 with a challenge body.
Pro-tip: whichever pattern you choose, tie rotation to outcome, not to a counter. A pool that rotates every N requests regardless of response codes will happily burn through good endpoints. Rotate on ban signals, keep the working session as long as it works.
Where the IP comes from matters more than the middleware
You can write flawless scrapy rotating proxies logic and still get 40% success rate, because anti-bot vendors classify the IP before your headers are ever parsed. DataDome, Cloudflare Bot Management, Akamai and PerimeterX pull ASN type and fraud score from IP intelligence databases such as MaxMind, IPQualityScore or Spur.us. The verdict is made on ASN ownership.
| Parameter | Datacenter | Mobile (MNO) |
|---|
| ASN type | hosting / business | mobile |
| Typical fraud score | 75–100 | 0–15 |
| Block risk | High | Minimal (CGNAT effect) |
| Speed | Very high | Medium, 5–50 Mbit/s |
| Success rate on hard targets | Low | 95–99% |
The reason mobile IPs survive is structural, not cosmetic. Carriers run Carrier-Grade NAT: one public IPv4 address is shared by 500 to 5000 subscribers at the same time. Banning it means banning thousands of paying customers of a real mobile operator, so platforms downgrade the response to a captcha or a rate limit instead of a hard block. That trade-off cannot be removed without the whole industry migrating to IPv6.
Second bonus: rotation on a mobile network is native behavior. A new IP after a cell handover or a PDP context reset is what every real smartphone does all day. Compare that to a datacenter subnet where three hundred sequential addresses hit the same endpoint within a minute.
Connecting OnlineProxy to your spider
Our endpoints are mobile ports of real cellular operators, delivered as host:port with login:password or IP whitelist authentication, over HTTP(S) or SOCKS5. For Scrapy you normally use the HTTP(S) endpoint; SOCKS5 is there when your stack also drives Playwright or an antidetect browser.
A minimal integration example looks like this: keep one credentials line per port in a plain text file, load it in your middleware, and assign the value to request.meta['proxy'] in the format http://login:password@host:port. Nothing exotic — the same string works in Selenium, Playwright, Colly or a bash curl check.
Rotation control depends on the plan:
- Lite — a shared port, up to five users per device, automatic IP change every 2–5 minutes. You cannot steer it, and device reboot is unavailable. Good for wide, forgiving crawls where a mid-session IP change costs nothing.
- Regular — the whole device is yours for the rental period. Sticky session, change by link, change by timer, device reboot, priority support. This is what you want when your spider logs in, keeps a cart, paginates a search result, or scrapes mobile SERP where the sequence must stay on one identity.
Billing is per port and per period — 1, 7 or 30 days, with 24 hours as the minimum billable period. There is no gigabyte metering on any plan, which is a different economics from per-GB residential pricing: a heavy crawl does not inflate your bill. Prices depend on country and operator and are shown on the tariff page. A single server proxy is available free through the widget on the site; free mobile ports and free mobile trials do not exist here.
Tuning Scrapy for a mobile channel
Mobile bandwidth is finite per device, typically 1–5 comfortable parallel streams. Copying a datacenter config onto a mobile port is the most common mistake we see.
- Drop CONCURRENT_REQUESTS_PER_IP to 2–4 and let AUTOTHROTTLE_ENABLED do the pacing.
- Raise DOWNLOAD_TIMEOUT to 60–90 seconds; latency of 50–300 ms plus a cell handover is normal, not a failure.
- Set RETRY_TIMES to 3 and add 429 to RETRY_HTTP_CODES, then force a fresh endpoint on retry.
- Keep geo consistency: an operator in Germany with Accept-Language set to Russian is a correlation red flag.
- Verify what you bought — check any port through Spur.us or IPQualityScore and confirm the ASN reads as mobile, not hosting.
And remember the boundary: scrapy rotating proxies solve the network layer only. TLS/JA3 fingerprints, headless browser artifacts and robotic timing patterns are separate problems, handled by curl_cffi-style TLS impersonation, an antidetect browser, and randomized delays.
Support and money
Support works around the clock with a 4-hour target for the first reply. Completed rentals generate cashback as promo credits on your internal balance. If a port has a technical problem we offer a replacement first; refunds follow the published refund and replacement policy — full within the first hour after access is issued, then minus the time already used.