How to Scrape Indeed (Python Guide)

Indeed is the largest job aggregator on the web, and its listings, salaries, and company data are the raw material for recruiting tools, labor-market research, and salary benchmarks. This guide builds a real Python scraper and tests it against the live site. I ran the fetch in June 2026, and Indeed blocked it. Below is the honest result, exactly what blocks you, and the route that actually returns job data.

Can you scrape Indeed, and is it legal?

You can send a request to Indeed, and in my test the public job pages refuse it, so “can you scrape it” splits into a technical answer and a legal one. The technical answer is below: a plain request gets a 403. The legal answer has two layers worth separating.

On US computer-access law, scraping pages anyone can load without logging in sits on the safer side. In hiQ Labs v. LinkedIn, the Ninth Circuit held that scraping public profiles does not violate the Computer Fraud and Abuse Act, and the Supreme Court’s Van Buren decision narrowed the CFAA toward traditional hacking. Reading a public job listing with a bot is not, by itself, a federal computer crime.

Two things still raise the risk for Indeed specifically. First, the contract layer: using the site means accepting Indeed’s terms, which restrict automated collection, and hiQ is the cautionary tale, it won the CFAA point and still paid a $500,000 judgment for breaching LinkedIn’s user agreement. Second, robots.txt. I fetched Indeed’s robots.txt (it returns 200) and it contains explicit bare directives:

Disallow: /jobs
Disallow: /viewjob
Disallow: /m/jobs
Disallow: /m/viewjob

So the search results (/jobs) and the individual listing pages (/viewjob) are declared off-limits to compliant crawlers. Ignoring robots.txt is not a crime on its own, and it is the kind of fact a court weighs in a terms dispute. Stay off logged-in pages, avoid collecting personal data (which pulls in the GDPR in Europe), and get legal advice before any commercial scrape. My is web scraping legal pillar covers the full picture.

What data is available on an Indeed job page?

A job listing carries most of the fields recruiting and research teams want, spread across the search results page and the individual job page. The catch in my test is that none of it is reachable with a plain request, because the page never loads. Here is what the public HTML exposes when you can render it, and where each field lives:

The job key (jk) is the field that matters most for a pipeline, because it is the stable identifier you dedupe and re-crawl on. Everything else hangs off it. Note that the company profile pages (/cmp/*, /company/*) are also restricted in robots.txt, so company enrichment is a separate, harder problem.

How does Indeed block scrapers?

Indeed sits behind Cloudflare, and a plain request from a datacenter IP gets a security challenge instead of job HTML. I tested this directly. Here is the exact request I sent:

import requests

headers = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/124.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}

url = "https://www.indeed.com/jobs?q=python+developer&l=Remote"
resp = requests.get(url, headers=headers, timeout=25)
print("STATUS:", resp.status_code)
print("server:", resp.headers.get("server"))
print("title:", resp.text[resp.text.find("<title>")+7 : resp.text.find("</title>")])

The real output:

STATUS: 403
server: cloudflare
title: Security Check - Indeed.com

Read that carefully, because it is the whole story for DIY. The request did not return a partial page or a missing field. It returned HTTP 403 with a Cloudflare-served “Security Check - Indeed.com” interstitial, 59,679 bytes of challenge HTML containing the strings cloudflare, captcha, and additional verification. The individual listing endpoint /viewjob?jk=... returned the same 403. The only Indeed URL that answered me with a 200 was robots.txt. These are the layers a DIY Indeed scraper runs into:

The honest summary: unlike a site that returns a 200 with a few missing fields, Indeed refuses the connection at the edge. A real browser User-Agent is not enough, because Cloudflare also fingerprints the TLS handshake that the requests library produces. The general playbook for getting past edge defenses is in my scraping without getting blocked guide; the Indeed-specific reality is that you need a real browser and a residential IP, which is what Method 2 supplies.

Method 1: scrape Indeed with requests and BeautifulSoup

A requests GET plus BeautifulSoup parsing is the standard DIY scraper, and on Indeed it does not get past the front door, so I am showing the code as the shape you would use if the page rendered, with the honest caveat up top. The parsing logic is correct; the fetch is what fails.

import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/124.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}

url = "https://www.indeed.com/jobs?q=python+developer&l=Remote"
resp = requests.get(url, headers=HEADERS, timeout=25)

# In my test this is 403, not 200. Guard for it so you do not
# parse a Cloudflare challenge page and log empty rows.
if resp.status_code != 200 or "Security Check" in resp.text:
    raise SystemExit(f"Blocked: HTTP {resp.status_code} (Cloudflare challenge)")

soup = BeautifulSoup(resp.text, "html.parser")

jobs = []
for card in soup.select("div.job_seen_beacon"):
    title_el = card.select_one("h2.jobTitle span[title]")
    company_el = card.select_one("[data-testid='company-name']")
    location_el = card.select_one("[data-testid='text-location']")
    link_el = card.select_one("a.jcs-JobTitle")
    jobs.append({
        "title": title_el.get("title") if title_el else None,
        "company": company_el.get_text(strip=True) if company_el else None,
        "location": location_el.get_text(strip=True) if location_el else None,
        "jk": link_el.get("data-jk") if link_el else None,
    })

print(f"parsed {len(jobs)} jobs")

I will not paste output for this block, because there is none to paste: the script exits at the guard with Blocked: HTTP 403 (Cloudflare challenge), which is the real behavior I observed. Faking a list of jobs here would be lying about a test, and the point of this site is that I do not do that.

Two routes can sometimes get a DIY scraper past the wall, with caveats. A real headless browser (Playwright) carries a genuine browser TLS fingerprint and runs the JavaScript, which clears the basic challenge more often than requests does. From a single residential IP at low volume, that can work for a few pages. It still breaks at scale, because Cloudflare escalates to interactive CAPTCHAs once it sees a pattern, and you are back to solving challenges by hand. For a one-off study of ten listings, a logged-in browser session you drive manually is the pragmatic answer. For anything repeatable, you need rotating residential IPs and managed challenge-solving, which is Method 2.

Method 2: scrape Indeed with a scraper API

A scraper API solves the two hard problems at once: it routes the request through a rotating residential IP and clears the Cloudflare challenge with a real browser, so you get job HTML back instead of a 403. You hand it the Indeed URL and parse the result with the same selectors as Method 1.

ChocoData is the service I use for this category. It exposes a universal endpoint (one URL for any site) plus 453 dedicated endpoints for common targets. Indeed does not have its own named dedicated endpoint in the docs, so the correct route is the universal endpoint with the full Indeed URL, which is what that endpoint is built for. Here is the request shape, swapping the raw requests.get for the API:

import requests
from bs4 import BeautifulSoup

API_KEY = "cd_live_YOUR_KEY"  # free tier: 1,000 requests/month, no card
target = "https://www.indeed.com/jobs?q=python+developer&l=Remote"

endpoint = "https://api.chocodata.com/api/v1/universal/get"
params = {
    "api_key": API_KEY,
    "url": target,
    "render": "true",   # run a real browser to clear the challenge
    "country": "us",    # US residential IP so US results resolve
}

resp = requests.get(endpoint, params=params, timeout=90)
print("STATUS:", resp.status_code)

# The rendered job HTML comes back in the payload; parse the same
# selectors as Method 1, but now you get listings instead of a 403.
soup = BeautifulSoup(resp.text, "html.parser")
print("cards:", len(soup.select("div.job_seen_beacon")))

The difference from Method 1 is what happens behind that one call. ChocoData fetches through a country-matched residential IP that rotates automatically and retries through a fresh IP when a challenge appears, and render=true runs a real browser so the Cloudflare interstitial that gave me a 403 gets cleared instead of returned. The country=us parameter gives Indeed a US context so the US results resolve. Billing applies only to successful (2xx) responses, so a blocked attempt does not quietly drain your budget.

I did not paste live API output here, because that requires a paid key and I will not fake numbers. The endpoint path, parameters, free-tier limit, and success-only billing above come from the ChocoData docs. The honest claim is narrow: the universal endpoint takes the Indeed URL and returns the rendered page through a residential IP, which removes the exact failure (HTTP 403 at the Cloudflare edge) I reproduced in my own test.

Should you scrape Indeed yourself or use an API?

Use a manually driven browser for a handful of listings, and a scraper API the moment you need volume or reliability, because plain DIY does not clear Indeed’s edge at all. The decision comes down to how many pages you pull and how often.

My honest take from the test: plain requests scraping of Indeed does not work, full stop, because the page never renders behind Cloudflare. A headless browser on a single residential IP is the cheapest thing that returns data, and it is fragile and does not scale. If you need Indeed job data reliably or in volume, a scraper API is the route that actually returns the page. And before you build anything, check whether Indeed’s official partner/OAuth access fits your use case, since the sanctioned route avoids the legal and robots.txt issues entirely.

To go deeper on the building blocks, see the Python web scraping guide for the language fundamentals, BeautifulSoup for parsing, and Scrapy if you are crawling many pages and want a framework. The web scraping pillar ties the whole workflow together.