A Scrapy proxy layer that keeps the crawl alive
A spider that works on ten pages and dies on ten thousand is rarely a code problem. Amazon, Google SERP, Booking and LinkedIn stop returning a usable response long before the parsing logic breaks: 403s, captcha pages, truncated HTML. A Scrapy proxy built on real mobile-carrier IPs shifts that fight away from the crawler — requests arrive from a cellular ASN, and the data lands in the item pipeline instead of the retry queue.
The numbers behind that are blunt. A datacenter IP typically scores 75–100 on fraud-score checks, a carrier IP sits at 0–15, and on hard targets where per-GB residential pools drop below 80% success, mobile ports hold 95–99%. Fewer retries per thousand pages means shorter crawl windows and a dataset you can actually deliver on schedule.
| IP source | How the target reads it | Success on protected web targets |
|---|
| Hosting / cloud | hosting ASN, high risk | Low, dies after the first few hundred requests |
| Home ISP pools | isp ASN, medium risk | Often under 80% on aggressive sites |
| Mobile operator (our ports) | mobile ASN, highest trust of all proxy types | 95–99% on the same targets |
One structural detail is worth naming once: a single carrier IP is shared by hundreds or thousands of real subscribers, so platforms answer with a captcha or rate limit instead of a hard ban. Practically, that means a rough hour of scraping costs you a slowdown, not a burned pool.
Scrapy http proxy: wiring takes minutes, not a sprint
Nothing exotic is required. Scrapy already ships HttpProxyMiddleware, so a scrapy http proxy is applied by putting the endpoint into request.meta in ip:port form with login and password — no extra pip install, no rewrite of existing spiders. Credentials stay in one settings file, and the same project you debugged locally goes to production untouched.
When routing needs to be selective, a small downloader_middleware of your own decides which port serves which domain: one function, a dozen lines of code, and a list of ports read from a file. Because billing is per port rather than per gigabyte, raising CONCURRENT_REQUESTS on the same endpoint never turns into a surprise invoice.
Authorization works both ways — login:password from anywhere, or IP whitelisting for a fixed server. HTTP(S) and SOCKS5 are both available, which matters when the crawler shares infrastructure with an antidetect browser profile.
Scrapy rotating proxies or one sticky IP?
Rotation is not a feature to enable everywhere. Sequential crawls that carry cookies, a logged-in session or pagination state break the moment the IP jumps mid-flow; mass collection of independent pages benefits from a fresh IP as often as possible. Choosing the right mode per spider is what turns scrapy rotating proxies from a buzzword into a lower block rate.
| Mode | Behavior | Where it pays off |
|---|
| Sticky session | IP held for the session you set | Logins, cart flows, paginated crawls with state |
| Rotation by link or API | New IP on demand from your own code | Change IP inside an errback after a captcha response |
| Timer rotation | Automatic change at a set interval | Overnight jobs left unattended |
| Shared-port auto rotation | IP changes every 2–5 minutes, not controllable | Cheap bulk collection where session state is irrelevant |
Pro-tip: tie rotation to outcomes, not to a schedule. Detect the captcha page in the parse callback, fire the rotation link, then reschedule the same request — the crawl repairs itself and you stop paying compute time for requests that were doomed anyway.
Retries, callbacks and concurrency that respect the modem
A mobile channel is not a datacenter uplink: expect 50–300 ms latency, 5–50 Mbit/s, and roughly 1–5 healthy parallel streams per device. Push 64 concurrent requests through a single port and the timeouts you see are your own doing, not the target's defense.
The fix is boring and effective. Keep AUTOTHROTTLE on, set a modest DOWNLOAD_DELAY, cap concurrency per domain, and route every failure into an errback that distinguishes a network drop from a block — retrying a 403 twenty times only teaches the target your pattern. With that in place, throughput comes from adding ports in parallel, which scales linearly and predictably.
Long crawls also survive better when the device can be reset. On a dedicated port a reboot is available, so a stuck modem is a two-minute interruption instead of a support ticket and a dead night of scraping.
Lite or Regular: which port to rent for a spider
| Parameter | Lite | Regular |
|---|
| Device access | Shared, up to 5 users | Dedicated for the whole rental |
| IP control | Auto rotation every 2–5 min | Sticky, by link, by timer |
| Device reboot | Not available | Available |
| Support | Standard | Priority |
For stateless page harvesting — price lists, catalogs, SERP snapshots — a Lite port is usually enough, and the forced rotation actually helps. Anything with a login, a multi-step form or an ordered pagination sequence belongs on Regular, where the session lasts exactly as long as your spider needs it to.
Unlimited traffic applies to both plans. Read that as no gigabyte metering rather than unlimited bandwidth: a 40 GB HTML harvest costs the same as a 2 GB one, which removes the most common budget overrun in web scraping projects.
Geo and operator targeting for correct data
Scraped data is only as good as the location it was collected from. Country, region, city and operator can all be pinned, so a rank check from Berlin on a specific carrier returns the real mobile SERP instead of a blended average. Clients get numbers that match what their customers actually see.
Geo consistency is the other half. If the port is German, the Accept-Language header, timezone logic and currency parsing should agree with it — mismatched signals are the fastest way to get soft-blocked even on a clean carrier IP.
Pro-tip: verify the port before the crawl, not after. A quick check through IPQualityScore or Spur.us should show a mobile ASN and a fraud score under 25. If a provider's IP resolves to hosting, you are paying mobile rates for datacenter behavior.
Common mistakes that cost crawl budget
- Running one port at datacenter concurrency, then blaming the target for timeouts.
- Rotating on a timer during a stateful flow, which drops half the parsed sessions.
- Ignoring IPv6 on carriers that hand it out — some tools and targets still handle it poorly, so confirm the address family your stack expects.
- Skipping request headers and TLS realism: the network identity is solved by the port, the browser identity is not.
What it costs and what happens if something breaks
A scrapy proxy here is rented per port for 1 day, 7 days or 30 days, with 24 hours as the minimum billing period; prices depend on country and operator and are shown live on the tariff page. There is no per-gigabyte line item, so the crawl cost is known before the spider starts — the only variable is how many ports run in parallel.
Testing the plumbing costs nothing: one server proxy is issued free through the widget on the site, which is enough to confirm that middleware, auth and callbacks behave. Mobile ports have no free tier, but completed rentals return cashback as promo credits to the internal balance, and the refund and replacement policy covers a full return within the first hour after access is issued, a pro-rated one later, and a replacement port first when the issue is technical.
Support runs around the clock with a target first response of 4 hours, and Regular ports go through the priority queue. For a data team on a deadline, that is the difference between a paused pipeline and a delivered dataset.