How to Scrape Yellow Pages (Python Guide)

by Marcus Reed · Published July 19, 2026 · 8 min read

Marcus Reed

Founder & lead tester · about the author

the short version

I fetched a live Yellow Pages search and listing URL on June 12, 2026 with requests: both returned HTTP 403 from Cloudflare, not the results HTML.
YP's robots.txt allows / for a generic agent but blocks named AI crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended), and sets Content-Signal: ai-train=no.
The useful fields (business name, phone, street address, rating, review count) sit in schema.org JSON-LD and class-based cards, so once you clear the wall, parsing is easy.
Lead with a scraper API (ChocoData example) that rotates IPs and clears the Cloudflare challenge, or run a headed browser with residential proxies.

Yellow Pages (yellowpages.com) still indexes millions of US businesses with phone, address, category, and rating, which makes it a steady source for lead lists, local SEO research, and market sizing. So “how do I scrape it” is a fair question. I built a Python scraper and pointed it at live YP on June 12, 2026. The honest headline first: a plain requests call does not get the page. YP answers with an HTTP 403 from Cloudflare, an anti-bot wall, before you see a single listing. This guide shows the real block, the parser that works once you are past it, and the two routes that actually return data. For fundamentals, see my web scraping guide and the Python walkthrough.

Can you scrape Yellow Pages, and is it legal?

You can extract public YP data, but a naive script is blocked, and the legality depends on method and what you keep. The technical wall comes first, covered in the next section. The legal picture has three layers worth separating.

On US computer-access law, reading pages anyone can load without logging in sits on the safer side. In hiQ Labs v. LinkedIn, the Ninth Circuit held that scraping public data does not violate the Computer Fraud and Abuse Act, and the Supreme Court’s Van Buren decision narrowed the CFAA toward traditional hacking. Querying a public YP listing with a bot is not, by itself, a federal computer crime.

Contract and robots terms are the second layer. YP’s Terms of Use restrict automated collection, and its robots.txt (full read below) grants a generic agent Allow: / while explicitly blocking named AI crawlers and attaching Content-Signal: search=yes,ai-train=no. That signal is an express reservation of rights: the operator permits indexing for search but refuses use of the content to train AI models. hiQ is the cautionary tale here: it won the CFAA question and still paid a $500,000 judgment for breaching LinkedIn’s user agreement.

Privacy is the third layer. Business names and addresses are low-risk, but YP listings often carry an owner’s personal phone or a sole-proprietor name, which is personal data under the GDPR and similar laws. Three rules keep you on the lower-risk path: stay on public business fields, never script a logged-in or paid account, and do not repurpose the data to train a model given the ai-train=no signal. I am a tester, not a lawyer, so get advice before any commercial scrape. My is web scraping legal pillar has the full picture.

What data can you extract from Yellow Pages?

A YP listing carries most of the commercially useful fields as schema.org JSON-LD, with the search-results page mirroring the same fields in class-based HTML cards. Knowing where each field lives tells you what to target.

Field	Where it lives	Easy to parse?
Business name	JSON-LD (`name`) + `a.business-name`	Yes
Phone	JSON-LD (`telephone`) + `div.phones`	Yes
Street address	JSON-LD (`address.streetAddress`) + `.street-address`	Yes
City / state / ZIP	JSON-LD (`address.*`) + `.locality`	Yes
Average rating	JSON-LD (`aggregateRating.ratingValue`)	Yes (when present)
Review count	JSON-LD (`aggregateRating.reviewCount`)	Yes (when present)
Categories	Listing page JSON / `.categories`	Inconsistent
Website link	Listing page (`.track-visit-website`)	Yes (when present)
Years in business	Listing page text	Inconsistent
Full review text	Listing page reviews block / paginated	Separate parse

The pattern: the headline fields ship both as a clean JSON-LD <script> block and as repeated div.result cards on the /search page, so once you have the HTML you parse a dict or a handful of CSS selectors, not a maze of brittle XPaths. Ratings and review counts appear on businesses that have them; treat those fields as optional in your code.

How does Yellow Pages block scrapers?

YP fronts its pages with Cloudflare, and I confirmed the block on live URLs. Here is exactly what I ran and got back on June 12, 2026:

import requests

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) "
                  "Chrome/124.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
}
url = ("https://www.yellowpages.com/search"
       "?search_terms=plumber&geo_location_terms=Los+Angeles%2C+CA")
r = requests.get(url, headers=headers, timeout=30)
print("STATUS", r.status_code)
print("SERVER", r.headers.get("Server"))
print("SNIPPET", r.text[:160])

The real response, not a guess:

STATUS 403
SERVER cloudflare
SNIPPET <!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
...
<title>Attention Required! | Cloudflare</title>

The city-path listing URL (/los-angeles-ca/plumber) returned the identical 403. Four things define the wall:

Cloudflare edge block. The Server header reads cloudflare and the body is a Cloudflare “Attention Required!” interstitial (~5.9 KB), not the listings. The 403 fires before YP’s app even runs.
TLS and header fingerprinting. Cloudflare inspects the TLS handshake (JA3/JA4) and header order, so a Python requests client is flagged even with a browser User-Agent. I swapped the UA and still got 403.
robots.txt signals. A generic agent gets Allow: /, but YP names and blocks AI crawlers and sets Content-Signal: search=yes,ai-train=no. So crawling for a search index is invited; taking the content to train a model is refused in writing.
JS challenge. Clearing the interstitial needs a real JavaScript engine to run Cloudflare’s script and return a token. requests cannot do that.

For reference, here is the part of YP’s live robots.txt that governs this, fetched the same day (it returned 200):

User-agent: *
Content-Signal: search=yes,ai-train=no
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Because the live fetch of the listings returned 403, I did not get results HTML to parse, so this guide carries no “tested” stamp and no fake listing output. The parsers below are real, working code that I verified locally against YP-shaped HTML, but they run against output you get past the wall (a scraper API’s HTML or a saved page), not against a raw requests call to YP.

Method 1: the scraper API route (recommended)

A scraper API is the route that returns YP data today, because it clears the Cloudflare challenge, rotates IPs, and renders JavaScript, then hands you HTML. You make one HTTP call; the vendor owns the wall.

ChocoData (chocodata.com) is one example I checked against its own docs. It exposes a universal endpoint (“one endpoint, any site”) plus 453 dedicated endpoints. The documented request shape, fetching a YP search page through the universal endpoint and parsing the result cards:

import requests
from bs4 import BeautifulSoup

API_KEY = "YOUR_API_KEY"  # free tier: 1,000 requests/month, no card

target = ("https://www.yellowpages.com/search"
          "?search_terms=plumber&geo_location_terms=Los+Angeles%2C+CA")

resp = requests.get(
    "https://chocodata.com/api/v1/universal",
    params={
        "url": target,
        "render_js": "true",   # run the Cloudflare challenge
        "api_key": API_KEY,
    },
    timeout=60,
)
resp.raise_for_status()
html = resp.text  # rendered HTML, past the 403

# YP repeats each business as a div.result card on the search page.
soup = BeautifulSoup(html, "html.parser")
rows = []
for card in soup.select("div.result"):
    name = card.select_one("a.business-name")
    phone = card.select_one("div.phones")
    street = card.select_one(".street-address")
    locality = card.select_one(".locality")
    rows.append({
        "name": name.get_text(strip=True) if name else None,
        "phone": phone.get_text(strip=True) if phone else None,
        "street": street.get_text(strip=True) if street else None,
        "locality": locality.get_text(strip=True) if locality else None,
    })

for r in rows:
    print(r)

For a single business detail page, parse the JSON-LD block instead, which is cleaner than selectors:

import json
from bs4 import BeautifulSoup

def parse_listing(html):
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup.find_all("script", type="application/ld+json"):
        try:
            data = json.loads(tag.string or "")
        except json.JSONDecodeError:
            continue
        items = data if isinstance(data, list) else [data]
        for d in items:
            if not isinstance(d, dict):
                continue
            if d.get("@type") in ("LocalBusiness", "Organization") or "aggregateRating" in d:
                addr = d.get("address", {}) or {}
                rating = d.get("aggregateRating", {}) or {}
                return {
                    "name": d.get("name"),
                    "phone": d.get("telephone"),
                    "street": addr.get("streetAddress"),
                    "city": addr.get("addressLocality"),
                    "region": addr.get("addressRegion"),
                    "zip": addr.get("postalCode"),
                    "rating": rating.get("ratingValue"),
                    "reviews": rating.get("reviewCount"),
                }
    return None

The endpoint pattern is GET /api/v1/{site}/{resource}?api_key=... for dedicated targets, plus the universal endpoint above for any URL. Per ChocoData’s site, the service handles residential proxy rotation, CAPTCHA and Cloudflare challenge solving, retries, and JavaScript rendering, and returns the rendered page. I have not independently benchmarked their YP success rate, so treat the parameter names (render_js, the endpoint path) as illustrative and verify against their docs on the free tier before you build on them.

I verified both parsing blocks above against YP-shaped JSON-LD and YP-shaped div.result markup locally, so the parsing logic is correct working code. What I did not do is run them end to end against live YP, because the raw fetch is walled. That is the honest line: the parsers are real, the live data depends on getting past Cloudflare, which is what the API does.

ChocoData’s published pricing: a free tier of 1,000 requests/month with no card, paid plans from $19/month (27,000 requests), and pay-as-you-go top-ups at $0.90 per 1,000 successful requests.

Method 2: DIY with a headed browser and proxies

If you want to scrape YP yourself, you need a real browser engine plus residential proxies, because Cloudflare blocks plain HTTP clients. Playwright drives Chromium, runs the challenge script, and renders the page; residential IPs keep you off the datacenter blocklists Cloudflare leans on.

# pip install playwright && playwright install chromium
from playwright.sync_api import sync_playwright

URL = ("https://www.yellowpages.com/search"
       "?search_terms=plumber&geo_location_terms=Los+Angeles%2C+CA")

def fetch_html(url):
    with sync_playwright() as p:
        browser = p.chromium.launch(
            headless=False,  # headless is flagged faster; headed clears more challenges
            proxy={"server": "http://USER:PASS@residential-endpoint:PORT"},
        )
        page = browser.new_page()
        page.goto(url, wait_until="networkidle", timeout=60000)
        html = page.content()
        browser.close()
        return html

# Feed html into the same div.result / JSON-LD parse from Method 1.

I did not run this end to end here, because it needs a paid residential proxy endpoint and a headed display, and faking the output would break the one rule that matters. Three honest caveats from how Cloudflare behaves:

Headless gets flagged faster. Cloudflare scores headless Chromium harder than a headed window, so headless=False (or a stealth plugin) clears more challenges, at the cost of needing a display or virtual framebuffer.
Datacenter IPs fail. A cloud server’s IP is on Cloudflare’s radar within a few requests. Residential or mobile proxies are effectively required, and they are the real cost line.
One browser per page is slow. Playwright spends seconds per page launching and rendering. For more than a few hundred listings, the API route is cheaper than the proxy plus compute bill.

For pagination, append &page=2, &page=3 to the search URL and loop until a page returns zero div.result cards. For the browser-automation fundamentals this builds on, see scraping without getting blocked. For the HTML-parsing side, BeautifulSoup covers the JSON-LD and selector extraction used above, and if you outgrow a single script, Scrapy handles queueing, retries, and the page loop.

Which method should you choose?

Pick based on volume, your tolerance for maintaining a proxy stack, and risk. Here is my decision rule.

Factor	DIY requests	DIY (Playwright + proxies)	Scraper API
Gets past Cloudflare	No (403)	You build it	Included
Returns data today	No	Yes (with residential IPs)	Yes (vendor-handled)
Handles JS challenge	No	Yes	Yes
Proxies / CAPTCHA	You own it (and still blocked)	You own it	Included
Volume ceiling	None reached (blocked)	Your proxy budget	Per request
Maintenance	High, breaks on every wall change	High	Low

Just exploring or a one-off lookup: open the page in a normal browser and copy what you need. A scripted requests call returns 403, so do not build on it.
High volume and you do not want to run infrastructure: use a scraper API. It removes the Cloudflare, proxy, and CAPTCHA line items for one per-request price, and the parsing code above plugs straight into its HTML.
You specifically want to own the stack and already have residential proxies: drive Playwright headed. Accept the per-page latency and the proxy bill.

The deciding fact from my run: a plain requests call to Yellow Pages returns 403 from Cloudflare, not data. Closing the gap to real listings means letting something (an API or your own browser-plus-proxy rig) run the JavaScript challenge and rotate IPs. For most readers past a handful of lookups, that is work a scraper API already did.

FAQ

Does Yellow Pages have an official API instead of scraping?

No public self-serve data API. YP markets advertising and a 'YP for Developers'-style listings feed to business partners, but there is no documented free endpoint that returns search results as JSON the way Yelp Fusion does. For programmatic access you either negotiate a data deal or scrape the public pages.

Why does my requests call to Yellow Pages return 403?

YP sits behind Cloudflare. A plain request gets a 403 with a Cloudflare 'Attention Required!' challenge body and a cf-managed cookie, not the listings. Cloudflare fingerprints the TLS handshake (JA3/JA4) and header order, so swapping the User-Agent alone does not clear it.

Yellow Pages vs Yelp vs Google Maps for local-business data, which scrapes easiest?

All three wall plain HTTP clients (YP and Yelp with edge anti-bot, Maps with JS rendering). YP and Yelp expose clean JSON-LD per business once you are past the wall; Maps needs a rendered front end or a SERP API. For a US business directory specifically, YP and Yelp overlap heavily, so pick by which one ranks the verticals you need.

Marcus Reed

I've built and run web scrapers for the better part of a decade. On this site I put scraper APIs and scraping tools through real jobs against real targets, then write up what actually holds up.

How I test · Methodology