Rotating proxies for web scraping without the retry tax
A crawler's real cost is not the proxy line in the invoice — it is the share of requests that come back as a captcha, a 403 or an empty template. Run a parser through hosting IPs and a large part of the budget quietly pays for retries instead of data. Carrier IPs hold the highest trust score of all proxy types, so a bigger share of the traffic you pay for returns rows that end up in the database.
What follows is a recipe, not a lecture: which rotation mode fits which target, how the pool behaves under load, and where the money leaks. No breakdown of anti-bot internals — only the settings that move success rate from "unstable" to predictable.
Why carrier IPs pay for themselves on protected targets
Google SERP, Amazon, Booking, LinkedIn and airline search treat hosting IPs as guilty by default. When residential pools fall below 80% success on those targets, mobile IPs typically hold 95–99% on the same code and the same headers. The difference shows up as records collected per hour of crawler uptime, not as a nicer graph in a dashboard.
One carrier IP sits behind hundreds or thousands of real subscribers, so a platform that bans it bans its own paying customers. The usual answer is a soft measure — a captcha or a rate limit — which a scraper survives with a pause and a fresh IP. Blanket IP bans stop being the thing that kills a nightly job at scale.
| Proxy type | IP origin | Risk on protected sites | Typical billing |
|---|
| Datacenter | Hosting and cloud ranges | High, fraud score 75–100 | Cheap per IP |
| Residential | Home ISP lines | Medium | Usually per GB |
| Mobile (OnlineProxy) | Real cellular carrier networks | Minimal, fraud score 0–15 | Per port for a period |
Choose the rotation mode by the job, not by habit
Scraping is not one task. A price list of 40,000 URLs, a login-protected dashboard and an overnight SERP crawl need different behavior from the same pool of ips. Matching the mode to the job is the cheapest optimization on the table.
| Mode | What it gives | Where it wins |
|---|
| Sticky session | Same IP held for a set number of minutes | Logged-in sessions, paginated catalogs, checkout flows |
| Rotation by link or API | New IP on command from your own code | Batch collection, recovery after a captcha |
| Timer rotation | Automatic change on a chosen interval | Long unattended runs overnight |
| Automatic 2–5 min (Lite) | IP changes on its own, no manual control | Mass parsing of light pages |
Rotation triggered from the scraper itself is the mode most teams underuse. Catch a 403 or a captcha marker, call the rotation link, repeat the request — and the job keeps running without a human on standby. One example: a Scrapy retry middleware that fires rotation after two consecutive failures usually rescues a run that used to die at 2 a.m. and cost a full morning of re-crawling.
Pro tip: rotate on failure, not on schedule. Rotation in the middle of a healthy session breaks pagination and cookies, and you end up paying twice for pages already collected. Timers are for long idle runs; link rotation is for errors.
Wiring it into Scrapy, Playwright or Go
Connection is the boring part, which is exactly how it should be. Credentials come as ip:port:login:password over HTTP(S) or SOCKS5, so Scrapy, Selenium, Playwright, Puppeteer and Colly accept them with one line in the settings file — no custom transport layer, no rewrite of working code. IP whitelisting is available too, which keeps passwords out of the repository on server-side jobs.
SOCKS5 matters beyond plain HTML fetching: it carries UDP, so headless sessions and antidetect profiles behave the way a real handset does. Teams that run a parser and a set of accounts side by side cover both usage patterns with one port type. Fewer moving parts means fewer places where a real IP can leak.
Real-time control comes from the API: current IP, remaining rental time, forced rotation, available geos. Proxy management becomes part of the pipeline instead of manual clicking in a panel between crawls.
Ports instead of gigabytes: where the scraping budget stops leaking
Per-GB pricing punishes exactly what a scraper does — pulling HTML in volume. OnlineProxy bills per port for a period of 1 day, 7 days or 30 days, with 24 hours as the minimum billing window, and no plan meters gigabytes. Unlimited traffic here means no per-GB invoice rather than unlimited bandwidth, which is the honest version of that promise.
The practical effect is simple: a heavy crawl costs the same as a light one, so depth and frequency can grow without a new spend forecast every morning. Prices depend on country and carrier and appear automatically on the plan page. After a paid rental ends, cashback arrives as promo credits on the internal balance, trimming the cost of the next port.
If a port has a technical problem, support offers a replacement first, and refunds follow the published refund and replacement policy — in full within the first hour after access is issued, later minus the time already used. Testing a city or a carrier for a single day before committing to a month costs one day. Worth knowing in advance: the free option in the site widget is one server proxy, not a mobile port, so rotating proxies for web scraping are always a paid rental here.
Pool size is a marketing number
A pool advertised in millions of ips usually points to an SDK model built on other people's phones, with unpredictable uptime and an unknown history behind each address. A provider that names specific carriers and cities and changes IP in two to five seconds is running physical modems with real SIM cards. For scraping, the second option means the geo in your report matches the geo you paid for.
Lite or Regular for a crawler
| Plan | IP control | Best fit |
|---|
| Lite | Shared device, up to 5 users, automatic rotation every 2–5 minutes | Stateless mass collection where any fresh IP works |
| Regular | Dedicated device for the whole rental: sticky sessions, rotation by link and timer, device reboot, priority support | Logged-in targets, SERP tracking, chained requests |
Rule of thumb: sessions require Regular, stateless volume runs cheaper on Lite. Mixed pipelines often use both — Lite for discovery of URLs, Regular for detail pages behind a login.
Geo and carrier targeting for data that matches reality
Prices, rankings and ad placements differ by city and by network. Selecting country, region, city and operator returns the mobile SERP a local subscriber actually sees, which is the difference between a report a client trusts and one they argue about. Ad verification gains the same way: the creative is checked as it renders inside the target network, so cloaking and misplacement surface before the budget is gone.
What carrier IPs do not fix
Network identity is one layer of four, and skipping the rest wastes good ips.
- Fingerprint: one port, one antidetect profile, one account — Multilogin, GoLogin, AdsPower, Dolphin Anty or Octo all take the credentials as they are.
- Pacing: randomized delays beat maximum threads, and a single modem comfortably holds roughly 1–5 parallel streams.
- Consistency: language, timezone and currency have to match the geo of the IP, otherwise the cleanest address still looks wrong.
- Verification: check a new IP through IPQualityScore or Spur.us and keep fraud score under 25 before the crawler starts.
Pro tip: write the IP into every scraped row. When a target starts answering with 429s, the log shows whether the cause was your code, your pace or one specific subnet — a five-minute diagnosis instead of a blind re-run.
Before you scale the farm
A short pass over these points prevents most of the money burned in the first month of a scraping project.
- Confirm the IP resolves to a cellular carrier, not to hosting.
- Match rotation mode to task before raising concurrency.
- Rent one port for a day in the target geo, measure success rate, then commit for 30.
- Keep retry logic and rotation calls in the same place in code.
- Support is available around the clock, with a target first response time of 4 hours — useful when a run stalls at night.
Handled this way, rotating proxies for web scraping stop being a line of risk in the plan and become a fixed cost per port with a predictable output in rows per hour. Carrier IPs remove the network-level blocks; antidetect profiles, sane pacing and consistent geo data close the rest. That combination is what keeps a scraping operation running while competitors rebuild their pipelines after every wave of bans.