Why a big crawl dies halfway through
The SEO Spider is fast, and that speed is exactly what gets it noticed. Ten threads against a mid-size store from one office address, and somewhere around the five-thousandth URL the responses turn into 429s, 403s and empty bodies. The report comes out half finished, the client waits another day, and the hours already spent on the crawl are written off.
The nastier scenario is not a block at all. Some sites keep answering, but serve a stripped or cached version to addresses they consider suspicious — and the audit then reports missing titles and thin content that real visitors never see. Fixing phantom issues costs far more than a proxy port, which is why the origin of the IP matters more than the raw speed of the channel.
Datacenter ranges are the first thing to be throttled, and per-gigabyte residential plans quietly drain the budget once JavaScript rendering is switched on. A carrier IP sits in the opposite corner: typical fraud score of a mobile address is 0–15 out of 100, against 75–100 for a hosting one. That gap is the difference between a crawl that finishes tonight and a crawl you restart three times.
What a mobile IP changes for the SEO Spider
Requests leave through the address of a real cellular operator, so the target treats the crawler as one more phone on the network. Because a single carrier address is shared by hundreds of subscribers at once, aggressive sites lean on soft measures instead of hard bans — the crawl slows down at worst, rather than dying. On targets where residential pools drop below 80% success rate, a mobile port usually keeps 95–99% of URLs answering, and that is the number that decides whether the report is publishable.
There is a second, purely SEO benefit. Mobile-first indexing means the version Google actually evaluates is the mobile one, and plenty of sites still ship different markup, different internal links and different meta tags to smartphones. Crawling from a carrier address with a smartphone user-agent shows that version directly, so the audit reflects the pages that earn rankings instead of a desktop copy nobody indexes.
Pro-tip: run the same section twice — once with the default agent from a regular connection, once with a smartphone agent through a mobile port — and diff the two exports. Every mismatch in canonicals, hreflang or meta robots is a real indexing bug you can bill for, found in one extra hour of machine time.
How to set up a Screaming Frog proxy on Windows, macOS and Ubuntu
The setup is the same on all three builds, and it takes about two minutes. The tool stores proxy details globally, not per crawl, so once it is configured every project inherits it until you change it back.
- Open Configuration → System → Proxy in the SEO Spider.
- Enter the host and the HTTP port of your mobile port.
- Restart the application — on Windows and macOS alike the setting is picked up only after a relaunch.
- Switch to List mode and crawl an IP-echo URL to confirm the exit address.
- Check that address once through an IP intelligence service: the ASN type must read as a carrier, not hosting.
That last step is the one most people skip, and it is the cheapest insurance in the whole stack. If a provider sells "mobile" but the check returns a hosting ASN, the crawl will be throttled exactly like before and the money is gone. Verifying takes thirty seconds and saves the whole rental period.
Important detail: the proxy fields in the Spider accept an address and a port, but no login and password. Trying to force login:pass into the host line simply breaks the connection. Use IP whitelisting instead — bind your working address once on the provider side, and the crawler connects with no credentials at all. Both authentication methods are supported here, so login:password stays available for Python, Scrapy or an antidetect browser on the same port.
One more compatibility note: the SEO Spider speaks HTTP(S) to a proxy and will not use SOCKS5. A port that offers both protocols covers the whole toolchain — HTTP for the crawler, SOCKS5 for antidetect profiles — so there is no need to rent two different products for one project.
Tuning speed so the crawl actually finishes
A cellular channel gives 50–300 ms of latency and 5–50 Mbit/s, which is plenty for HTML but unforgiving of a 20-thread stampede. A single device comfortably handles a handful of parallel streams; push past that and the timeouts you see are your own, not the site's. Dialing the settings down once means the crawl runs unattended overnight instead of being babysat.
| Setting in the Spider | Value for a mobile port | What it buys |
| Speed → Max Threads | 2–5 | No self-inflicted timeouts, stable response codes across the whole crawl |
| Speed → Max URI/s | 2–3 | The pace looks like browsing, so rate limiting never kicks in |
| System → Response Timeout | 30–60 s | Slow pages are recorded, not dropped as errors |
| System → 5xx Retries | 3 | Occasional network hiccups get retried instead of polluting the report |
JavaScript rendering multiplies traffic by an order of magnitude, because every page pulls scripts, fonts and images. On per-gigabyte billing that is where the invoice explodes; on a port rented for a period the volume simply does not enter the calculation. Rendering a 50,000-URL web project costs the same as crawling raw HTML.
Rotation: keep one IP or change it on schedule
The crawler holds a single proxy for the entire session, so a pool that rotates on every request is the wrong tool — mid-crawl address changes produce inconsistent responses and broken sessions on staging environments behind a login. What works is a dedicated port whose IP you control yourself.
| Mode | When it wins | Plan |
| Sticky session | Crawling behind authentication, comparing two runs on the same address | Regular |
| Change by link | A 403 wall appeared — new IP on demand, then resume the crawl | Regular |
| Timer rotation | Long overnight crawls of hundreds of thousands of URLs | Regular |
| Automatic every 2–5 min | Short scans, position checks, one-off list crawls | Lite |
For SERP work the logic flips. Checking positions in Google from a specific city and carrier benefits from frequent changes, because each request then looks like a different subscriber in the same location — and the ranking data reflects the local mobile results rather than a personalized desktop page.
Lite or Regular for crawling work
Lite is a shared port: one device serves up to five users, the IP rotates automatically every 2–5 minutes, and neither the rotation nor a device reboot is under your control. For a quick audit of a few thousand URLs that is enough, and it is the cheaper entry point.
Regular hands over the whole device for the rental period. Sticky sessions, rotation by link, rotation by timer, a reboot button and priority support come with it — everything a multi-day crawl of a large web property needs. Agencies running client audits back to back tend to keep one Regular port per active project, so a stalled crawl on one site never touches another.
What it costs and how billing works
Billing here is per port for a period: one day, seven days or thirty days, with 24 hours as the minimum. Unlimited traffic is included on both plans, which means gigabytes are never metered — not that the cellular channel becomes infinitely fast. Prices depend on the country and the carrier and are shown on the tariff page, so the cost of a crawl is known before it starts.
After a paid rental ends, cashback is credited to the internal balance as promo credits, which offsets the next project. If a port turns out to be technically faulty, a replacement is offered first, and refunds follow the refund and replacement policy — inside the first hour after access is issued the payment is returned in full, later minus the time already used. Support works around the clock with a four-hour target for the first reply, so a broken crawl on a Sunday deadline is not left waiting until Monday.
Free access covers one server proxy from the widget on the site, useful for a smoke test of the setup path. Mobile ports are paid from the first day — worth knowing before planning a trial.
Mistakes that waste a rental
- Forgetting the restart. The Spider ignores new proxy details until it relaunches, and the first "clean" crawl goes out from the office address.
- Geo mismatch. A German port with a US locale and timezone in the crawl setup is an instant red flag; keep country, language and agent consistent.
- One port for everything. A Screaming Frog proxy shared with antidetect profiles links the sessions together, and a warning earned by the crawler lands on the accounts.
- Maximum threads by default. Twenty streams through one device produce a report full of fake timeouts.
- Skipping the ASN check. Thirty seconds of verification prevents paying for a hosting address dressed as a carrier one.
None of this replaces the rest of the stack. A mobile address solves the network layer; browser fingerprints, request pacing and geo consistency stay your responsibility, and for account-related work an antidetect browser remains part of the setup. For crawling, though, the network layer is where 90% of failures live.
Set the port once, cap the threads, verify the exit address, and the audit finishes on the first attempt. A screaming frog proxy on a carrier IP does not make the crawler faster — it makes the crawl complete, the data honest, and the invoice predictable.