How to Scrape Job Postings (Python Guide)

Job postings are one of the most requested scraping targets I get asked about: salary benchmarks, hiring-trend dashboards, lead lists for recruiters, and “who is hiring for X” trackers all start with listing data. The catch is that job boards vary wildly in how they treat scrapers. Some block a script at the first request; one of the most popular remote boards hands you a clean JSON API. This guide builds a real Python job scraper, runs it against live boards, and pastes the actual output, including the 403s. I ran every command below in June 2026. Then I cover the route that works when a board blocks you.

Can you scrape job postings, and is it legal?

You can scrape job postings from public pages, and reading public listings sits on the safer side of US law, but most boards’ terms ban automated access and that is where the real risk lives. It helps to separate three layers.

On US computer-access law, fetching pages anyone can load without logging in is broadly defensible. In hiQ Labs v. LinkedIn, the Ninth Circuit held that scraping public profiles does not violate the Computer Fraud and Abuse Act, and the Supreme Court’s Van Buren decision narrowed the CFAA toward genuine hacking. So requesting a public job listing with a bot is not, by itself, a federal computer crime.

Contract law is the live constraint. Most boards’ terms of service prohibit bots and automated access, and that is a breach-of-contract question, not a criminal one. The data type matters too: a job title, salary range, and location are facts about the role and carry low privacy risk. A named recruiter, a contact email, or anything tied to an identifiable person is personal data, which pulls in the GDPR in Europe and the CCPA in California. Stay on public listing pages, never script a logged-in account, avoid collecting personal data, and honor each board’s robots.txt. I am a tester, not a lawyer, so get advice before a commercial scrape. My is web scraping legal pillar covers the full picture.

What data is on a job posting?

A listing page and a search results page together carry most of the commercially useful fields. Here is what is typically available and where it sits, assuming you can retrieve the page:

Title, company, location, and date are the fields you can rely on. Salary is the prize and the most often missing: many employers leave it blank, so treat pay coverage as partial on every board.

How do job boards block scrapers?

The large aggregators block in two layers: an anti-bot wall on the request, and JavaScript hydration on the page. I probed four boards with a plain requests GET and a real browser User-Agent. Here is the exact code:

import requests

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/124.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}

boards = {
    "Indeed":       "https://www.indeed.com/jobs?q=python+developer&l=Remote",
    "Glassdoor":    "https://www.glassdoor.com/Job/python-developer-jobs-SRCH_KO0,16.htm",
    "ZipRecruiter": "https://www.ziprecruiter.com/jobs-search?search=python+developer",
    "RemoteOK":     "https://remoteok.com/remote-python-jobs",
}

for name, url in boards.items():
    r = requests.get(url, headers=HEADERS, timeout=20)
    print(f"{name:13} {r.status_code}  {len(r.content):>7} bytes")

This is the real output I got:

Indeed        403    59637 bytes
Glassdoor     200   983676 bytes
ZipRecruiter  403     5915 bytes
RemoteOK      200    60672 bytes

Indeed and ZipRecruiter returned 403 on every run. I fetched Indeed’s page title and it reads “Security Check - Indeed.com”, an anti-bot interstitial rather than listings. Glassdoor returned a 200 here, but on a different run the same URL came back 403, and the 983 KB body is a consent/challenge shell, not parsed listings, so I do not trust it as a stable path. RemoteOK returned a clean 200.

The detection stack on the big boards has a few layers:

• TLS and HTTP/2 fingerprinting. A real Chrome connection has a specific TLS handshake (JA3/JA4) and frame order that requests cannot fully imitate, so the board can flag a bot before reading a header.
• IP reputation. Datacenter and cloud IPs, where your script runs, are scored as high-risk.
• JavaScript hydration. Even on a 200, aggregators inject listings into the DOM after load. A static HTML parse sees the shell, not the jobs. I confirmed this on RemoteOK’s HTML page: the tr.job rows came back without the title and company attributes a browser would render.

Matching the User-Agent solves none of this. For the building blocks behind getting past it, see how to scrape without getting blocked.

Method 1: scrape a public job API with Python (RemoteOK)

When a board offers an API, use it: it returns clean structured data and skips the anti-bot wall entirely. RemoteOK publishes a free JSON feed at remoteok.com/api. The first element of the array is a legal notice, then each remaining element is a job object. Here is the exact code I ran:

import requests

HEADERS = {"User-Agent": "bestscraperapi-demo/1.0 (+https://bestscraperapi.com)"}

resp = requests.get("https://remoteok.com/api", headers=HEADERS, timeout=30)
print("STATUS:", resp.status_code)

records = resp.json()
jobs = records[1:]                 # records[0] is RemoteOK's legal/attribution notice
print("JOBS  :", len(jobs))

for job in jobs[:5]:
    title   = job.get("position")
    company = job.get("company")
    loc     = job.get("location") or "n/a"
    smin    = job.get("salary_min")
    smax    = job.get("salary_max")
    pay     = f"${smin:,}-${smax:,}" if smin and smax else "n/a"
    print(f"- {title} | {company} | {loc} | {pay}")

This is the real output I got, live:

STATUS: 200
JOBS  : 100
- Rural and Suburban Mail Carrier | Canada Post / Postes Canada | Saint Vincent's-Saint Stephen's-Peter's River,  | n/a
- Director Operations Logistics | Cloetta | New York, New York, United States | n/a
- sdf | Women Builders Council | n/a | $60,000-$90,000
- asdffd | Women Builders Council | n/a | $60,000-$90,000
- sadfsdf | Women Builders Council | n/a | $60,000-$90,000

The call returned 100 job objects with the fields parsed cleanly. Two honest notes from the live data. First, several rows from one poster are obvious test junk (sdf, asdffd), so a real pipeline needs a validation step: drop listings with sub-three-character titles or known placeholder text. Second, salary is sparse: most rows came back n/a because the employer left pay blank, which matches the coverage caveat from the data table above.

Each job object carries more than I printed. The full key set is:

slug, id, epoch, date, company, company_logo, position,
tags, description, location, apply_url, salary_min,
salary_max, logo, url

One condition you must honor. RemoteOK’s API returns its terms in that first array element, and they are explicit: link back to RemoteOK with a normal follow link (no nofollow) and name RemoteOK as the source, or they suspend API access. That is a fair trade for a free, structured feed, so build the attribution link into whatever you publish. If you are new to the parsing side, the Python web scraping guide and requests walkthroughs cover the basics.

What about LinkedIn’s guest endpoint?

LinkedIn’s logged-out jobs search at /jobs/search returned a 200 with about 60 job cards in the HTML when I tested it, which is more than Indeed gives you. I am not building a method on it here, because that endpoint is rate-limited hard, shifts its markup often, and scraping it runs straight into LinkedIn’s terms, the same terms at issue in hiQ. I cover the specifics in the LinkedIn scraping guide. Treat it as fragile, not a stable source.

Method 2: scrape job boards with a scraper API

For boards that block, like Indeed and ZipRecruiter, a scraper API gets past the 403 that stopped Method 1. It runs your request through a real browser engine and a rotating residential IP, so the board sees a browser-shaped connection instead of a Python script, and it renders the JavaScript so the listings are actually in the HTML you get back.

ChocoData is the service I use for this category. It exposes a universal endpoint (one URL for any site) plus 453 dedicated endpoints for common targets. You hand it the board URL and it returns the rendered page. Here is the request shape, swapping the bare requests.get from the block probe for the API:

import requests
from bs4 import BeautifulSoup

API_KEY = "cd_live_YOUR_KEY"  # free tier: 1,000 requests/month, no card
target  = "https://www.indeed.com/jobs?q=python+developer&l=Remote"

endpoint = "https://api.chocodata.com/api/v1/universal/get"
params = {
    "api_key": API_KEY,
    "url": target,
    "render": "true",   # run a real browser engine so Indeed sees a browser
    "country": "us",    # US residential IP to match the board's region
}

resp = requests.get(endpoint, params=params, timeout=90)
print("STATUS:", resp.status_code)

# Rendered HTML comes back in the payload; parse the job cards with
# BeautifulSoup exactly as you would a page you fetched directly.
soup = BeautifulSoup(resp.text, "html.parser")
cards = soup.select("[data-testid='job-card'], .job_seen_beacon")
print("CARDS :", len(cards))

The difference from Method 1 is what happens behind that one call. ChocoData fetches through a country-matched residential IP that rotates automatically and retries through a fresh IP when a block appears, and the render flag runs a browser engine so the page’s JavaScript executes and the listings land in the HTML. Per ChocoData’s docs, billing applies only to successful (2xx) responses, so the 403 wall from Method 1 does not quietly drain your budget. I did not paste live API output here because it requires a paid key, and I will not fake numbers. The honest claim is narrow: the universal endpoint takes a URL and returns the rendered page, which removes the two failures I reproduced (the 403 block and the empty JS shell). I have not independently benchmarked their success rate on Indeed, so treat the parsing claims as vendor-stated until you test on the free tier.

Which method should you use?

Use an official or public API where the board offers one, and a scraper API for boards that block. The choice comes down to whether the board hands you data willingly.

My honest take from the test: there is no single “job scraper” that works everywhere, because boards split into two camps. RemoteOK and other API-publishing boards give you structured data for free if you attribute them, so always check for an API first. Indeed, Glassdoor, and ZipRecruiter block a plain script and render with JavaScript, so a scraper API is the practical way to get a real response instead of the 403 I hit.

To go deeper on the tooling, see the Python web scraping guide for the language fundamentals, BeautifulSoup for parsing HTML, and Scrapy if you are crawling many boards and want a framework. The web scraping pillar ties the whole workflow together.