How to Scrape Craigslist (Python Guide)

Craigslist looks like an easy scrape: plain HTML, no login wall on listings, decades of classifieds across housing, jobs, for-sale, and services. The catch is not the markup, it is the law. Craigslist has won a CFAA case against a scraper and collected a seven-figure settlement, which makes it one of the riskier targets to scrape commercially. This guide builds a working Python scraper, runs it against a live San Francisco rentals page, shows the real output, and is honest about the legal line and the result cap. I ran the code in June 2026 and pasted the actual results below. Then I cover the API route for when you need volume.

Can you scrape Craigslist, and is it legal?

You can technically fetch and parse public Craigslist pages, and I did exactly that below, but Craigslist has a stronger legal record against scrapers than almost any other site, so the answer is “yes, with real caution.” Two layers matter and they point in different directions here.

The general US rule on public data leans permissive. In hiQ Labs v. LinkedIn the Ninth Circuit held that scraping data anyone can view without logging in does not violate the Computer Fraud and Abuse Act, and the Supreme Court’s Van Buren decision narrowed the CFAA toward genuine unauthorized access. Loading a public listing page with a script is not, on its own, a federal computer crime.

Craigslist is the case that cuts the other way, so treat it as its own category. In Craigslist Inc. v. 3Taps Inc. (N.D. Cal. 2013), Judge Charles Breyer held that once Craigslist sent 3Taps a cease-and-desist letter and blocked its IP addresses, continuing to access the site through proxy servers violated the CFAA. The block plus the letter revoked authorization, and routing around them was the violation. 3Taps settled in 2015 and paid Craigslist $1,000,000. The practical lesson: a first scrape of public pages is low-risk, but if Craigslist blocks you and tells you to stop, scraping around that block is the act courts have penalized.

Contract law stacks on top. Craigslist’s Terms of Use are blunt: “You agree not to copy/collect CL content via robots, spiders, scripts, scrapers, crawlers, or any automated or manual equivalent (e.g., by hand).” They also bar making derivative works, distributing, or selling CL content other than postings you created yourself. The robots.txt disallows /reply, /flag, /mf, /mailflag, and a few posting-flow paths, so the reply and flag endpoints are explicitly off limits even before the broader terms.

My honest read: scraping a few public Craigslist pages for personal research sits in the same low-risk zone as any public page, but commercial scraping, redistribution, or scraping after a block is where Craigslist has actually sued and won. I am a tester, not a lawyer, so get advice before anything commercial. My legal overview covers the wider picture.

What data is available on a Craigslist listing?

A search results page gives you the core fields per listing, and the detail page adds structured attributes. Here is what I pulled in my run and the selector each field lives under:

The useful structural fact: Craigslist ships a no-JavaScript fallback inside the search HTML. The modern site renders results client-side, but the same page also contains a <li class="cl-static-search-result"> list for browsers without JS. That fallback is plain HTML you parse with BeautifulSoup, so you do not need a headless browser for the search results. The detail pages are server-rendered HTML too.

How does Craigslist block scrapers?

Craigslist does not run a heavy commercial bot wall like PerimeterX, so its defenses are lighter than Zillow’s or Amazon’s, but they are real. The signals it uses:

• IP-based rate limiting and bans. Craigslist watches request rate per IP and will return a 403 or a “blocked” page, and at the extreme it bans the IP. This is the exact mechanism from the 3Taps case: the block is the legal trigger, not just a technical one.
• The 360-result cap. Each search returns at most about 360 listings regardless of how many match. My SF rentals run returned 351. This is a soft anti-bulk-collection measure: you cannot pull a whole category in one query.
• Reply and contact gating. Phone numbers and emails sit behind a reply button that loads on click, often via a relay address. robots.txt disallows /reply outright.
• Datacenter IP suspicion. Cloud and datacenter ranges get throttled faster than residential ones, so a server doing repeated pulls trips limits sooner than a single home connection.

The honest framing from my own test is in the next section: from one clean IP at low volume, none of this stopped me. At scale, IP rate limiting is the wall you hit, and routing around a block is the line 3Taps drew.

Method 1: scrape Craigslist with requests and BeautifulSoup (what really happened)

A requests GET with a browser User-Agent plus BeautifulSoup to parse the cl-static-search-result list is the right first attempt, and it worked cleanly for me. Here is the exact code I ran against a live SF Bay apartments-for-rent search:

import csv
import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/124.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}

url = "https://sfbay.craigslist.org/search/apa"  # SF bay apartments for rent
resp = requests.get(url, headers=HEADERS, timeout=25)
print("STATUS:", resp.status_code)

soup = BeautifulSoup(resp.text, "html.parser")

# Craigslist embeds a no-JS fallback list in the search HTML.
rows = soup.select("li.cl-static-search-result")
print("RESULTS:", len(rows))

listings = []
for li in rows:
    a = li.find("a", href=True)
    title = li.get("title")
    price = li.find("div", class_="price")
    loc = li.find("div", class_="location")
    listings.append({
        "title": title,
        "price": price.get_text(strip=True) if price else "",
        "location": loc.get_text(strip=True) if loc else "",
        "url": a["href"] if a else "",
    })

with open("cl_listings.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=["title", "price", "location", "url"])
    w.writeheader()
    w.writerows(listings)

for x in listings[:5]:
    print(f'{x["price"]:>8} | {x["location"][:16]:<16} | {x["title"][:34]}')
print(f"Wrote {len(listings)} rows")

This is the real output I got (one representative run; I repeated the request five times in the same session and all five returned 351):

STATUS: 200
RESULTS: 351
  $3,945 | foster city      | Beautiful Waterfront Living, Walk-
  $5,608 | san mateo        | LED Lighting, Ground Floor Retail,
  $1,685 | concord / pleasa | Nice large new paint 1 bedroom apa
  $2,495 | hayward / castro | large 2 bedroom 2 bath townhouse a
  $4,295 | pacific heights  | Updated Studio Laundry In Unit Clo
Wrote 351 rows

Page one returned 351 listings with price, location, title, and detail URL, written straight to CSV. The parse is clean because the no-JS fallback is simple, flat HTML. I confirmed reliability by repeating the fetch 5 times back to back: all five returned HTTP 200 with 351 results.

To enrich a listing, fetch its detail URL and read the attribute spans. I ran this against the first result and it returned price $3,945, attributes ['2BR / 2Ba', '960ft2', 'w/d in unit', 'carport', 'no smoking'], and coordinates (37.5538, -122.270):

import requests
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
           "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"}

url = "https://sfbay.craigslist.org/pen/apa/d/san-mateo-beautiful-waterfront-living/7936152602.html"
soup = BeautifulSoup(requests.get(url, headers=HEADERS, timeout=25).text, "html.parser")

price = soup.select_one(".price")
attrs = [s.get_text(" ", strip=True) for s in soup.select(".attrgroup span")]
mp = soup.select_one("#map")

print("price:", price.get_text(strip=True) if price else None)
print("attrs:", attrs[:8])
print("geo  :", (mp.get("data-latitude"), mp.get("data-longitude")) if mp else None)

Now the honest caveats. First, the 360-result cap is hard: one search will not give you a whole category. Second, I ran this from a clean datacenter IP at low volume, and that is why it passed. Push the request rate up or run it from a flagged IP and Craigslist rate-limits you with a 403 or a block page. Third, and most important for Craigslist specifically, if it does block you, scraping around that block is the conduct that lost in 3Taps. I was not blocked in this short test, so I am reporting success, but I would not hammer it.

Paging and scaling the DIY version

Because each search caps near 360 results, you scale by slicing the query, not by paginating one search forever. Useful filters on the apartments search:

Run many narrow queries (per subregion, per price band), collect the posting ID from each detail URL, and dedupe across queries. This is the legitimate way around the cap. The moment you are firing dozens of these from one IP, you are back to the rate-limit and block problem, which is what Method 2 handles. Before going further, read how to scrape without getting blocked.

Method 2: scrape Craigslist with a scraper API

When narrow queries multiply and one IP starts getting throttled, a scraper API runs each request through a rotating residential IP so Craigslist sees ordinary-looking traffic and the block rate drops. ChocoData is the service I use for this category. It exposes a universal endpoint (one URL for any site) plus 453 dedicated endpoints across 235 sites. Craigslist is not one of the dedicated endpoints, so the universal endpoint is the route: you hand it a Craigslist URL and parse the returned HTML exactly as in Method 1.

import requests
from bs4 import BeautifulSoup

API_KEY = "cd_live_YOUR_KEY"  # free tier: 1,000 requests/month, no card
target = "https://sfbay.craigslist.org/search/apa?min_bedrooms=2&max_bedrooms=2"

endpoint = "https://api.chocodata.com/api/v1/universal/get"
params = {
    "api_key": API_KEY,
    "url": target,
    "parse": "html",   # raw rendered HTML; then BeautifulSoup as in Method 1
    "country": "us",   # US residential egress
}

resp = requests.get(endpoint, params=params, timeout=90)
print("STATUS:", resp.status_code)

# Same parser as Method 1 - only the fetch changed.
soup = BeautifulSoup(resp.text, "html.parser")
rows = soup.select("li.cl-static-search-result")
print("RESULTS:", len(rows))

The difference from Method 1 is what happens behind that one call. ChocoData fetches through a country-matched residential IP that rotates automatically and retries through a fresh IP when a block appears, which is the failure mode I flagged: a single IP that works until Craigslist rate-limits it. It also keeps you from the 3Taps trap of manually proxy-hopping around a ban you received directly. Per ChocoData’s docs, the free tier is 1,000 requests per month with no card, and a standard request is charged against successful responses, so a block does not quietly drain your budget. I did not paste live API output here because it requires a key, and I will not fake numbers. The endpoint path, parameters, and free-tier limits above come from the ChocoData docs. The honest claim is narrow: a rotating-residential API removes the single-IP rate-limit fragility of the DIY path; I have not independently benchmarked its Craigslist success rate, so test it on the free tier.

Which method should you use?

Use DIY for a small, one-off personal pull; use the API when you need many queries or reliability, and stop entirely if Craigslist blocks you and tells you to. Here is the trade-off from what I actually observed:

My honest take from the test: a plain Python scraper works on Craigslist today, the no-JS fallback makes parsing easy, and I got 351 real listings on a repeatable 200. The two catches are the 360-result cap (which forces query-slicing, not pagination) and the legal record. Craigslist is the rare site that has sued scrapers and won, so a low-volume personal pull is fine, but commercial scraping, redistribution, or scraping after a block is where the risk is concrete. For volume, route narrow queries through a scraper API so one rate-limited IP does not stall the job, and never proxy around a ban aimed at you.

To go deeper on the building blocks, see the Python web scraping guide for the language fundamentals, BeautifulSoup for parsing the listing HTML, and Scrapy if you are running many regional queries and want built-in throttling and dedupe. The web scraping pillar ties the whole workflow together.