How to Scrape Job Postings from Job Boards: Step-by-Step

What Scraping Job Postings Means and What You Can Use It For

To scrape job postings, collect public job listings from boards and careers pages. A scraper turns them into structured data rows: title, company, location, salary, posting date, apply URL, source URL and description. The real challenge is not downloading HTML. It is keeping those fields stable when layouts, selectors or ATS templates change. Most projects first crawl listing pages to build detail-page URLs, then extract each page separately.

Job board scraping turns public sources into records you can compare. Typical uses include:

• labor-market snapshots by role, seniority or location;
• salary benchmarking across companies;
• competitor hiring tracking or hiring signals for sales;
• aggregators that refresh listings daily.

Web scraping job postings works best when you need repeated collection. Public Greenhouse and Lever boards are common starting points. For example, an agency can pull every new Data Engineer opening from competitor careers pages into one spreadsheet each morning.

Is It Legal to Scrape Job Postings? What to Check First

Public visibility does not create a right to collect. Before you scrape job postings, read the site’s terms of service first and treat robots.txt as a policy signal, not a technical guarantee. Even polite crawling can conflict with contract rules. One German job-ad pipeline study (2026) followed robots.txt and request throttling. Yet StepStone’s terms could still be read to prohibit automated collection. Do not bypass login walls or CAPTCHAs. Recruiter names, applicant details and some salary records can be personal data under GDPR or CCPA, so exclude them unless you have a lawful basis. If a field identifies a recruiter, drop it. Prefer public company boards, official APIs or feeds, keep request volume low, and document your checks for each source. This is not legal advice. Consult counsel for your use case.

Choose Your Method: API, No-Code Tool or Your Own Scraper

Pick the method by the number of sources and the anti-bot pressure. For job board scraping, official ATS feeds are the safest first choice. Public Greenhouse, Lever and Ashby boards often return structured JSON, so you skip fragile HTML selectors. If a site has no feed, choose between a no-code scraper, a hosted scraping API, or a custom script.

The trade-off is clear: a no-code scraper is fast on day one, but you give up stability when markup changes. A custom script costs more setup and upkeep, yet it gives control over pagination, cleaning and alerts. To scrape job listings at scale, prefer feeds, then write code only for what the feed cannot provide.

How to Scrape Job Postings Step by Step

How to scrape job postings reliably: run the same five-step workflow on any listing source or careers page.

1. Define the schema: pick one board, roles, places, freshness, and required fields.
2. Find the data source: JSON-LD, JSON endpoint, or HTML.
3. Fetch and extract: request pages politely and map results into columns.
4. Clean and deduplicate: normalize salary, dates, locations, and repeated roles.
5. Export and schedule: save CSV or database rows and run it on a timer.

Pipeline: career page URL → JSON-LD or HTML → eight clean fields → CSV every morning.

Step 1: Define the Target and the Fields You Need

Before writing any scraper, decide exactly which job postings you need. Pick the boards or career pages, the roles, locations, and freshness window, such as the last seven days. Then fix a schema. A useful minimum is job ID, title, company, location, remote flag, salary range, posting date, description, apply URL, source URL, and collection timestamp. For example, if you track Product Designer roles in Berlin from the last seven days, every row should fill the same columns. This keeps results comparable across different boards. It also makes row validation and missing-field checks easier, preventing painful cleanup later.

Step 2: Find Where the Job Data Actually Lives

Before you scrape job listings, inspect the markup for JobPosting JSON-LD. Without structured data, Google can only guess whether a page is a job posting. For scraping, JSON-LD removes that ambiguity by giving title, location, datePosted and salary as machine-readable fields. Public ATS boards often expose clean structured feeds directly. If the grid loads dynamically, open DevTools, choose the Network tab, filter by Fetch/XHR, and look for a JSON endpoint returning the whole list. It is usually more stable than parsing rendered HTML.

The snippet below uses Python requests to pull a career page and reads JobPosting schema blocks from JSON-LD. If no such source exists, fall back to CSS selectors, but keep them narrow. Layout redesigns often break selector-based extraction, while structured fields change less often. Save the endpoint URL, request headers and required query parameters before writing the extractor. A reliable response reduces cleanup and makes validation easier.

import requests
import json
import re

page = requests.get(URL, headers={"User-Agent": "Mozilla/5.0"})
page.raise_for_status()

blocks = re.findall(
    r'<script[^>]*application/ld\+json[^>]*>(.*?)</script>',
    page.text,
    flags=re.S
)

for block in blocks:
    data = json.loads(block)
    items = data if isinstance(data, list) else [data]

    for item in items:
        if isinstance(item, dict) and item.get("@type") == "JobPosting":
            print(item.get("title"), item.get("datePosted"))

Step 3: Fetch the Pages and Extract the Fields

Fetch listing pages first, then open a detail page only when the list lacks salary, description, or apply URL. Prefer the JSON endpoint found earlier. Parse HTML only when no feed exists. In many developers’ experience, such feeds are faster and more reliable than automating dynamic pages. Use requests for static or JSON responses. Reserve Playwright or another headless browser for JavaScript-rendered boards, infinite scroll, or AJAX-loaded grids. Set a realistic User-Agent. Apply rate limiting with random delays and low concurrency. Throttling slows incoming requests to prevent overwhelmed servers, per Balancing Security and Privacy (2025). Add retry logic with a hard cap so one bad URL cannot loop forever. Extract only the schema fields, store source URL and timestamp, and flag rows with missing required values instead of saving them silently. For web scraping job postings, this keeps the run predictable and the output stable. The pattern below follows pagination links, saves each job as a dictionary, and pauses between attempts.

import time
import random
import requests

HEADERS = {
    "User-Agent": "USER_AGENT"
}

def fetch_jobs():
    rows = []

    # Read pages 1 through 10.
    for page in range(1, 11):
        url = f"https://HOST/api/jobs?page={page}"

        # Try each page up to three times.
        for attempt in range(3):
            response = requests.get(url, headers=HEADERS, timeout=20)

            if response.ok:
                data = response.json()

                for job in data.get("jobs", []):
                    rows.append({
                        "job_id": job["id"],
                        "title": job["title"],
                        "location": job.get("location"),
                        "apply_url": job.get("url"),
                        "source_url": url,
                    })
                break

            time.sleep(random.uniform(2, 4))

        # Pause before the next page.
        time.sleep(random.uniform(2, 4))

    return rows

Step 4: Clean, Normalize and Deduplicate the Data

Raw job listings need cleaning before analysis. Flatten nested salary and poster objects. Use HTML stripping for descriptions, then apply normalization to locations, currencies, and pay ranges. Convert relative dates such as “3 days ago” into absolute dates. Deduplication uses a job ID or canonical URL. If a board relies on session-specific links, fall back to company, title, and region. In U.S. employer postings, a 2020 study, Medicaid expansion job postings, removed repeats with the same role, company, and area inside a two-month window. Save source_url and a collected_at timestamp on every record. Run validation against the schema with Pydantic so missing fields fail loudly. For example, “80k to 95k USD” becomes salary_min and salary_max.

Step 5: Export and Schedule Your Job Scraper

Export each run to a durable format. CSV and JSON work for files. Google Sheets suits shared review. A database helps when you query history.

Schedule the scraper with cron, a scheduler, or a cloud job. A daily 7:00 run can append fresh rows and alert you if fewer than 10 arrive. Also record expired postings. Compare current job IDs with yesterday’s set. Mark disappeared roles as closed and log the row count. Send another notification after repeated failures or blocked requests. That makes a sudden zero visible immediately.

Scraping Specific Job Boards: Company Career Pages, Indeed and LinkedIn

For job board scraping, public company boards and ATS pages like Greenhouse or Lever are usually the most reliable starting point. Large aggregators such as Indeed and LinkedIn invest heavily in bot detection, and their terms of service often restrict automated access.

Public Company Boards and ATS Pages

Public careers subdomain pages from Greenhouse, Lever, and Ashby are the best first targets for job listings. They usually expose a clean JSON endpoint behind the listing grid, so parsing is more stable than scraping rendered HTML. Their bot pressure is low, and one schema can cover many employers with minimal changes. Since most companies do not sell a jobs API, these public boards often become the practical source.

Why Indeed and LinkedIn Are Harder Targets

Indeed and LinkedIn render job grids with JavaScript and tie results to location cookies, accounts, and session state. Simple requests often receive a login wall, CAPTCHA, or rate limit. Even an HTTP 200 can contain empty results, with no visible job rows. Their terms of service restrict automated access, so review them first. When the same role appears on the employer’s career page, use that source or an official product.

How to Scrape Job Postings with Proxies

To scrape job postings with proxies, route requests through an intermediary that conceals the original IP. Boards reject heavy traffic from one address through rate limits, reputation checks, or flagged datacenter ranges. IP rotation spreads the load. Geo-targeting can also match the city or country a board expects, because listings may change by location. One query may produce separate results for Berlin and Munich.

Choose session behavior by task. Rotating endpoints suit many independent searches. A sticky session keeps one address while following pagination, filters, or detail pages, so the site does not treat each fetch as a new visitor. For example, if collecting the same “Nurse” search in six cities, assign one persistent identity per city, add a three-second request delay, and cap retries.

Source category changes trust and maintenance. As Balancing Security and Privacy (2025) notes, datacenter addresses come from cloud providers, not ISPs. Residential alternatives route through real home connections and usually look more ordinary. Carrier-based options use telephone-network addresses. OnlineProxy serves mobile proxies through modems and SIM routers on real 4G/5G carrier IPs, so they resemble ordinary mobile-network traffic. They use username and password authentication and support HTTP or SOCKS5 in scraping scripts. A dedicated device with sticky sessions and IP change by link or timer fits paginated crawls.

Use this Python requests with a proxy pattern for an HTTP proxy:

import requests

proxies = {
    "http": "http://USERNAME:PASSWORD@HOST:PORT",
    "https": "http://USERNAME:PASSWORD@HOST:PORT",
}

response = requests.get("https://BOARD.example/jobs", proxies=proxies, timeout=20)
print(response.status_code)

For SOCKS5, install the extra requests[socks] support and change the scheme to socks5. A proxy lowers block risk, but it does not fix broken selectors, grant permission, or make restricted collection appropriate. Keep delays, respect terms, and verify the target before publishing.

Common Problems: Blocks, Empty Results and Broken Selectors

When you scrape job postings, treat failures as loud signals. HTTP 403 or HTTP 429 usually mean the board sees pressure. Slow the pace, add delays, rotate IPs and check headers. Balancing Security and Privacy (2025) notes that rule-based bot detection checks User-Agent, Accept-Language and Referer.

A CAPTCHA is a stop sign. Lower the request rate or change the source instead of building the pipeline around bypassing it. If the page returns HTTP 200 but zero rows, suspect JavaScript rendering, missing cookies or a regional layout. Use check your proxy's IP location to see what region the site sees.

Broken selectors are common after small page updates. Prefer JSON, JSON-LD or schema-based fields over brittle CSS paths. Deduplicate by job ID or canonical URL, and mark stale rows when postings disappear. Add monitoring: alert on zero-result runs and sudden row-count drops before they corrupt the dataset. A normal status code makes the failure easy to miss. For example, if a layout change leaves zero rows, the script can still log success. Two weeks later, a salary report may rely on stale listings.

What to Do with Scraped Job Data

Turn the rows into decisions, not archives. In Google Sheets, group postings by role, location and date to see labor market movement. Compare posted pay ranges for salary benchmarking, but normalize currency and benefits first. Create job alerts for roles that match your team or candidates.

For B2B outreach, treat new openings as hiring signals. A company hiring for a function you serve can be a warm prospect. Build lead lists only when the trigger is relevant, and respect privacy rules. For example, if an agency filters for Growth Marketing Manager in London, it can contact those employers with a tailored pitch. Drop posts older than 30 to 45 days unless they are reposted.

FAQ: Scraping Job Postings

Is it legal to scrape job postings?

It depends on the site’s terms of service, the data you collect, and local privacy law. Public pages can still be restricted. Check the terms first before scraping, avoid personal data and login-protected pages, and consult counsel for commercial use, especially for recurring, large-scale collection.

Can I scrape job postings from Indeed or LinkedIn?

You can technically attempt both, but they are hard targets. Indeed and LinkedIn use bot detection, login walls, CAPTCHAs, and may return empty results. LinkedIn’s user agreements explicitly prohibit automated scraping. Prefer official APIs, official products, or the employer’s own public career page when possible.

Do I need proxies to scrape job listings?

Usually, no. You can scrape job listings from a few public career pages without a proxy. Keep the request rate low. Add one for rate limits, frequent or large crawls, or geo-targeting for local results. On protected boards like Indeed, rotation, throttling and session reuse are not optional where allowed.

How often should I scrape job boards?

Match your schedule to how fast the source changes. For most career pages, scrape once daily. Daily runs suit alerts and fast-changing contract roles. Hourly runs are rarely needed and raise block risk. Use a time filter, such as last 24 hours, to fetch only new or updated roles.

Conclusion: A Reliable Job Scraping Workflow

A reliable workflow starts with public boards and structured JSON, fixes one schema, then extracts, cleans, deduplicates and adds monitoring. Pace requests, respect each site's terms of service, and use proxies only when volume, blocks, or location-specific results require them. For the first run, scrape job postings from one career page. Validate 20 rows against the schema, then schedule repeat runs and row-count alerts before expanding.