How to Scrape Crunchbase (Python Guide)

Crunchbase is the default reference for startup funding: rounds, investors, valuations, headcount, and the founders behind each company. That makes it a constant scraping target and a hard one. I spent June 2026 pointing a plain Python scraper at a live organization page. The short version: the page returned a 403 served by Cloudflare with an “Attention Required!” block, so requests plus BeautifulSoup never reached any company data. This guide shows exactly what I got, what data Crunchbase actually holds, how it blocks, and the two routes (official API or scraper API) that work when the naive approach does not.

Can you legally scrape Crunchbase?

Scraping public Crunchbase data sits in three legal buckets you weigh independently: US computer-crime law, Crunchbase’s terms of use, and data-protection law. All three matter here because Crunchbase mixes company facts with named people.

On US computer-crime law, the anchor case is hiQ Labs v. LinkedIn. On April 18, 2022, the Ninth Circuit ruled that scraping publicly available data likely does not violate the Computer Fraud and Abuse Act, because the CFAA’s “without authorization” bar does not reach data open to the public without a login. That precedent helps less than it looks here, because Crunchbase’s deeper data sits behind a login and credit wall, so it is not “open to the public without a login” in the first place, and the logged-out profile pages were blocked by Cloudflare in my test anyway.

On contract, Crunchbase restricts automated access and data reuse in its terms of use, and breaching those terms is a civil matter (account termination, breach claims) that is separate from the criminal CFAA question. I did not retrieve a verbatim anti-scraping clause from the live terms page during testing, so I am not quoting one here. Confirm the current wording on Crunchbase’s own terms page before you rely on it.

On data protection, the picture is sharp. Crunchbase profiles name founders, executives, and investors and attach roles and histories to them. That is personal data under GDPR and the CCPA even though it is business information. Scraping it makes you a separate data controller with your own obligations: a lawful basis, notice, and honoring deletion and objection requests. I cover the general framing in my is web scraping legal guide. For Crunchbase specifically, treat every person-level record as personal data and get counsel before processing identifiable EU, UK, or California residents at scale.

What data is available on Crunchbase?

Crunchbase holds deep firmographic, funding, and people data, but only a thin slice shows on a logged-out profile page and the valuable detail is gated by login and credits. Here is what each surface holds, based on the public page structure and Crunchbase’s own product descriptions:

The pattern: a company profile exists at a clean URL like /organization/<slug>, but the funding tables, investor lists, valuations, and people detail that make Crunchbase valuable render for logged-in accounts and often cost credits or a Pro subscription to reveal. Even the logged-out profile is shielded by Cloudflare, which is where my request stopped. The next section shows exactly how.

How does Crunchbase block scrapers?

Crunchbase fronts its site with Cloudflare, and a plain Python request to a public org page returned a hard 403 block. I confirmed this live in June 2026 with Python’s requests and a real browser User-Agent. Here is the probe I ran:

import requests

UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " \
     "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
headers = {
    "User-Agent": UA,
    "Accept-Language": "en-US,en;q=0.9",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
}

r = requests.get(
    "https://www.crunchbase.com/organization/openai",
    headers=headers, timeout=25,
)
print("status:", r.status_code)
print("server:", r.headers.get("Server"))
print("cf-ray:", r.headers.get("cf-ray"))
print("body bytes:", len(r.text))
print("'Attention Required' in body:", "Attention Required" in r.text)
print("'Cloudflare Ray ID' in body:", "Cloudflare Ray ID" in r.text)

This is the real output I got:

status: 403
server: cloudflare
cf-ray: a0aa26dc2c60e24d-VNO
body bytes: 5469
'Attention Required' in body: True
'Cloudflare Ray ID' in body: True

The 403 is served by Cloudflare, and the 5,469-byte body is Cloudflare’s “Attention Required!” block page carrying a Cloudflare Ray ID, not Crunchbase company data. A plain HTTP client from a datacenter IP gets fingerprinted and refused at the edge before any page HTML loads. Because I never received a Crunchbase page, there is nothing to parse, and I am not stamping this guide as a working tested scrape. Reaching the real profile needs a browser-grade fingerprint plus a clean residential IP.

I also pulled the robots.txt directly, which returned 200. Crunchbase disallows /search, /discover, /textsearch, /login, /register, /account, and /v4 (its API path) for the default user-agent, and it blocks a long list of AI and LLM crawlers outright, including GPTBot, CCBot, anthropic-ai, Claude-Web, PerplexityBot, and Google-Extended. It does publish a sitemap at https://www.crunchbase.com/www-sitemaps/sitemap-index.xml. robots.txt is not a legal contract, but it makes Crunchbase’s intent for the search and discovery surfaces explicit, so honor it and stay on individual profile URLs rather than hammering search endpoints.

Method 1: Can you scrape Crunchbase with Python directly?

Not with requests and BeautifulSoup, because the request is blocked with a 403 before any HTML arrives, as the output above shows. BeautifulSoup parses HTML you already have in hand, and a Cloudflare block page is the only HTML a plain client receives, so the parse step never gets real data to work with. If you read my Beautiful Soup guide, the parsing is the easy part; getting an unblocked response is the wall on Crunchbase.

The only self-hosted approach with any chance is a real browser engine via Playwright, which presents a browser fingerprint and runs whatever client-side checks Cloudflare serves. The skeleton looks like this:

# pip install playwright && playwright install chromium
from playwright.sync_api import sync_playwright

URL = "https://www.crunchbase.com/organization/openai"

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto(URL, wait_until="domcontentloaded", timeout=60000)
    # Cloudflare may need a few seconds to clear or to serve a block.
    page.wait_for_timeout(8000)
    title = page.title()
    print("Title:", title)
    if "Attention Required" in title or "Just a moment" in title:
        print("Still on the Cloudflare block / challenge page.")
    browser.close()

I am not pasting output for this snippet, because a default headless Chromium from a datacenter IP is exactly the fingerprint Cloudflare refuses, so an honest result here would just be another block. In practice, vanilla Playwright clears low-tier Cloudflare sometimes and fails the hard tier, and Crunchbase runs the hard tier. Making it work consistently means adding stealth patches, a residential IP, and retry logic, at which point you are rebuilding a scraper API by hand. For the techniques involved, see my guide on scraping without getting blocked. If you want the request-and-parse fundamentals first on sites that actually let you in, my Python web scraping guide covers the basics.

One more DIY caveat: even if you clear Cloudflare, the funding tables, investor lists, and people detail render for logged-in accounts and often cost credits, so the logged-out page gives you a slim profile at best. That gating is the real reason the official API or a scraper API beats a hand-rolled scraper here.

Method 2: How do you scrape Crunchbase with a scraper API?

Send the target profile URL to a scraper API with JavaScript rendering switched on, and it runs a real browser, rotates residential IPs, and returns the rendered HTML, so the Cloudflare 403 becomes the vendor’s problem instead of yours. Because the DIY path is blocked at the door, this is the realistic technical route for any public Crunchbase profile data.

ChocoData is one such service, with a universal endpoint plus a set of dedicated endpoints. The request shape is a single GET carrying your target URL and an API key, with JavaScript rendering on for a Cloudflare-protected target like Crunchbase:

import requests

API_KEY = "YOUR_API_KEY"
target = "https://www.crunchbase.com/organization/openai"

resp = requests.get(
    "https://api.chocodata.com/api/v1/universal",
    params={
        "api_key": API_KEY,
        "url": target,
        "render_js": "true",
    },
    timeout=120,
)
print(resp.status_code)
# resp.text now holds the rendered HTML; parse it with BeautifulSoup.

I did not run this snippet, because it needs a paid key, so treat the parameter names as illustrative and confirm them against ChocoData’s own docs before you build on it. The principle holds across any reputable scraper API: you hand it the URL, it returns the rendered page from a rotating residential IP, and the Cloudflare block, the JS rendering, and the rate limits become the vendor’s problem. ChocoData advertises residential proxy rotation and optional JavaScript rendering alongside 453 dedicated endpoints, which are the capabilities Crunchbase’s Cloudflare layer makes mandatory.

Once you have rendered HTML back, the parse is ordinary BeautifulSoup. A logged-out Crunchbase profile exposes its name and description in standard tags, so a first pass looks like this:

from bs4 import BeautifulSoup

soup = BeautifulSoup(resp.text, "html.parser")
name = soup.find("h1")
desc = soup.find("meta", attrs={"name": "description"})
print("Name:", name.get_text(strip=True) if name else None)
print("Description:", desc["content"] if desc else None)

Two limits a scraper API does not remove. First, the login and credit wall: it gets you past Cloudflare to the public profile, but it does not log you into a Crunchbase account, so the deep funding tables, valuations, and investor lists stay gated unless you supply an authenticated session, which is a more direct terms breach. Second, the legal layer is untouched. Routing through residential proxies grants you no permission under Crunchbase’s terms and no GDPR cover when you store founder or executive records. Use it for public, factual company data, keep person-level records out of your store unless you have a lawful basis, and get counsel before any collection touches identifiable EU, UK, or California residents.

Should you scrape Crunchbase or use the official API?

For Crunchbase specifically, the official API is usually the better answer, with scraping as a narrow fallback for public profile data only. Here is the honest trade-off across the three routes:

Crunchbase sells API access to its own database under contract, which returns structured JSON, keeps you inside the terms, and is the only clean way to reach the round-by-round funding and people data that make the product valuable. Confirm the current plans, fields, and rate limits on Crunchbase’s own data and API pages, since entitlements change and I am not quoting prices I did not verify live. Scraping earns its place only when you need public profile data at a volume or price the API does not serve, and you have weighed the contract and data-protection risk. Whichever route you pick, the obligations on personal data follow the founder and executive records wherever they go, so build opt-out and deletion handling in from the start rather than bolting it on later.