What a crawler proxy actually changes in your scraping economics
A spider that hits Google SERP, Amazon or Booking from a hosting range burns its budget on retries: a large share of calls come back as captcha walls, and the run still costs full time and compute. A crawler proxy built on real carrier IPs flips that ratio — targets that give under 80% success on per-GB residential pools typically return 95–99% on mobile IPs. Fewer retries mean shorter runs, smaller pools, and a dataset that closes tonight instead of next week.
The reason is structural, not clever: carrier addresses sit in mobile ASNs and are shared by hundreds of subscribers at once, so platforms answer them with a soft rate limit rather than a hard ban. For a scraping team that translates into one thing — the target server keeps returning 200 while datacenter ranges get walled off after the first few hundred calls.
Crawling proxy options compared: where the money leaks
| Parameter | Datacenter | Residential (ISP) | Mobile (ours) |
|---|
| ASN type | hosting | isp | mobile carrier |
| Typical fraud score | 75–100 | medium | 0–15 |
| Block risk on protected targets | High | Medium | Minimal |
| Billing | Per IP | Per GB | Per port, per period |
Cheap IPs look like savings until the success rate is factored in: a blocked request costs the same as a successful one. Choosing a carrier IP by country, city and operator also fixes SERP monitoring, where mobile results differ from desktop ones and only a real mobile network shows what your audience actually sees.
Crawlee proxy setup in a few lines of code
Crawlee, Scrapy, Playwright, Puppeteer and Colly all accept a proxy as a single connection string, so a crawlee proxy configuration is a few lines of code — feed ip:port:login:password into ProxyConfiguration and the request queue routes itself. HTTP(S) and SOCKS5 are both available, login:password auth works from any machine, and IP whitelisting fits a fixed office address. No SDK, no rewrite of an existing spider, no lost day of engineering time.
Rotation modes: choose by job, not by habit
- Sticky session — the address holds while an authorized crawl walks through pagination, so the session survives to the last page.
- Rotation by link — one HTTP call forces a new IP, letting the spider change identity exactly when a 429 shows up instead of on a blind schedule.
- Rotation by timer — set the interval once and overnight runs keep going without a babysitter.
- Automatic change every 2–5 minutes on Lite — no control over timing, but enough for stateless page harvesting.
Pro tip: before the first big run, check the port through Spur.us or IPQualityScore. Fraud score under 25 and a mobile ASN mean the pool is real; a "hosting" verdict means substitution, and your crawlers will hit the same walls you paid to avoid.
Lite or Regular for your crawlers
| Option | Lite | Regular |
|---|
| Device access | Shared, up to 5 users | Dedicated for the whole rental |
| IP control | Auto every 2–5 min, not adjustable | Sticky, by link, by timer |
| Device reboot | Unavailable | Available |
| Support | Standard | Priority |
Lite is the pragmatic pick for volume parsing where any fresh IP will do. Regular pays for itself the moment a job needs a held session, a forced change after a soft block, or a reboot at 3 a.m. — one dedicated device removes the neighbor risk that ruins long queues.
Pricing: ports and days, no gigabyte meter
Rental runs by port for 1, 7 or 30 days, with a 24-hour minimum period, and traffic is unlimited on both plans — meaning no per-GB billing, not unlimited bandwidth. For a crawl that pulls tens of gigabytes of HTML, that removes the scariest line in the estimate: the bill stops growing with every retry. Prices depend on country and carrier and are shown on the plan page; cashback arrives as promo credit on the internal balance after a paid rental ends, and refunds follow the refund and replacement policy — full within the first hour after access is issued, later minus the time used, with a replacement offered first if the cause is technical. Support works around the clock with a four-hour first-response target.
What a crawler proxy will not fix
Network identity is one layer of the stack. Fingerprint and behavior are separate jobs: pair a port with an antidetect browser such as Dolphin Anty, Octo or AdsPower for account-bound tasks, randomize delays, and keep language, timezone and user agent aligned with the proxy geo. Treat the port as one tool among four and success rates hold; skip the rest and even a clean carrier IP won't rescue a scraper firing forty pages a second.