How to Scrape Google Search (SERP) in Python

I tried to scrape Google’s search results page the simple way and pasted the real result below. Short version: a plain Python requests call gets HTTP 200 but no results, because Google returns a JavaScript bootstrap page to clients that do not run JS. This guide shows what the SERP holds, the exact failure you will hit, a working DIY attempt with Playwright, and the scraper-API path that returns clean JSON. For background on the fundamentals, see my web scraping guide and the Python scraping walkthrough.

Can you legally scrape Google search results?

Scraping public SERP data sits in a contested but commonly practiced gray area, and you should read three documents before you start. Google’s own robots.txt is the clearest signal: I fetched it on June 12, 2026 and it contains Disallow: /search with only narrow carve-outs.

Disallow: /search
Allow: /search/about
Allow: /search/howsearchworks
Allow: /search
Disallow: /shopping/search
Disallow: /travel/search

robots.txt is a crawling convention, not a law, and it does not by itself create legal liability. The legal questions come from elsewhere:

The hiQ ruling concerned public data and the Computer Fraud and Abuse Act, and it leaned toward scrapers for openly accessible pages. It did not bless breach-of-contract claims, and it is US-specific. My rule of thumb: scrape only public result pages, never log in, rate-limit hard, and keep personal data out of what you store. For deeper coverage of blocking and ethics, see scraping without getting blocked. None of this is legal advice; if the data feeds a commercial product, talk to a lawyer.

What data can you extract from a Google SERP?

A modern Google results page is a layout of distinct blocks, and each block carries different fields. Knowing the structure tells you what to target and what an API should return as JSON.

Two of these matter for method choice. People Also Ask and the knowledge panel are populated and expanded by JavaScript, so a static HTML fetch would miss them even if Google served you raw markup. That pushes any serious DIY attempt toward a real browser.

How does Google block scrapers?

Google blocks no-JS clients at the first request and escalates from there. Here is exactly what I observed and what comes next.

When I requested the search page with requests and a desktop Chrome user agent, the response was HTTP 200 but contained an enablejs redirect inside a <noscript> block, with CSS that hides every table, div, and span. There are no <h3> titles and no /url?q= result links to parse. The 200 is misleading: it is a bootstrap page, not the SERP.

If you push harder (more requests from one IP, obvious automation), Google escalates:

1. JavaScript gate (the enablejs interstitial) - hits no-JS clients immediately.
2. Rotating, obfuscated CSS class names - even with a real browser, selectors like div.g break across regions and over time.
3. IP rate-limiting - bursts from a single datacenter IP trigger throttling.
4. The /sorry/ CAPTCHA page - reCAPTCHA challenge once you look like a bot.
5. Region and language variance - results differ by gl, hl, and the IP’s location, so output is not stable without geo control.

So the defenses are layered: JS execution, then IP reputation, then a CAPTCHA wall. A scraper needs to answer all three.

Method 1: DIY scraping with Python (the honest result)

A plain requests + BeautifulSoup script returns HTTP 200 but zero results, and here is the code plus the real output. I ran this on June 12, 2026 with Python 3.13.7 and requests 2.34.2.

import requests
from bs4 import BeautifulSoup

headers = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/120.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}

url = "https://www.google.com/search?q=web+scraping+python&hl=en&gl=us"
r = requests.get(url, headers=headers, timeout=20)
print("STATUS:", r.status_code)

soup = BeautifulSoup(r.text, "html.parser")
print("h3 titles found:", len(soup.select("h3")))
print("result links found:", len(soup.select('a[href^="/url"]')))
print("enablejs gate present:", "enablejs" in r.text)

The real output:

STATUS: 200
h3 titles found: 0
result links found: 0
enablejs gate present: True

The 200 status fools beginners. The page is the JavaScript bootstrap, so there is nothing to parse. The fix that actually renders results is a headless browser. Playwright executes the JavaScript, so the DOM contains real result nodes. Install with pip install playwright then playwright install chromium.

from playwright.sync_api import sync_playwright

def scrape_serp(query: str):
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page(
            user_agent=(
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                "AppleWebKit/537.36 (KHTML, like Gecko) "
                "Chrome/120.0.0.0 Safari/537.36"
            ),
            locale="en-US",
        )
        page.goto(
            f"https://www.google.com/search?q={query}&hl=en&gl=us",
            wait_until="domcontentloaded",
        )
        # Consent screens appear in some regions; click through if present.
        try:
            page.click("button:has-text('Accept all')", timeout=3000)
        except Exception:
            pass

        results = []
        for block in page.query_selector_all("div.g, div[data-hveid]"):
            title_el = block.query_selector("h3")
            link_el = block.query_selector("a")
            if title_el and link_el:
                results.append({
                    "title": title_el.inner_text(),
                    "url": link_el.get_attribute("href"),
                })
        browser.close()
        return results

if __name__ == "__main__":
    for i, row in enumerate(scrape_serp("web+scraping+python"), 1):
        print(i, row["title"], "->", row["url"])

I am not pasting fake output for the Playwright version. On a fresh datacenter IP it frequently lands on a consent wall or the /sorry/ CAPTCHA, and the selectors (div.g) change across regions, so results are inconsistent run to run. That is the honest state of DIY Google scraping. To make it production-grade you would add: residential proxy rotation, per-region gl/hl handling, consent-screen logic, CAPTCHA solving, and selector maintenance every time Google reshuffles class names. For the browser-automation fundamentals see the Scrapy guide and BeautifulSoup guide.

DIY costs at scale add up fast:

Method 2: Scraping Google with a scraper API

A scraper API turns the whole problem into one HTTP call that returns JSON, and that is why most teams skip the browser stack. The API runs the headless browser, rotates residential IPs, solves CAPTCHAs, and parses the SERP server-side. You send a query, you get structured fields.

ChocoData (chocodata.com) is one example I checked against its own docs. It exposes a universal endpoint (“one endpoint, any site”) across 235 sites plus 453 dedicated endpoints, including a Google search endpoint. The documented request shape:

import requests

API_KEY = "YOUR_API_KEY"  # free tier: 1,000 requests/month, no card

resp = requests.get(
    "https://chocodata.com/api/v1/google/search",
    params={"q": "coffee maker", "api_key": API_KEY},
    timeout=60,
)
data = resp.json()  # JSON in, JSON out

for item in data.get("organic_results", []):
    print(item["position"], item["title"], item["url"])

The endpoint pattern is GET /api/v1/{site}/{resource}?api_key=..., so the same request style works for other targets by swapping the path. Per ChocoData’s site, the service handles residential proxy rotation, CAPTCHAs, failed-request retries, and optional JavaScript rendering, and returns “clean structured JSON.” I have not independently benchmarked their success rate, so treat their parsing claims as vendor-stated until you test on the free tier.

How the two methods compare on the things that actually matter:

ChocoData’s published pricing: free tier of 1,000 requests/month with no card, then paid plans from $19/month (27,000 requests) up, and pay-as-you-go top-ups at $0.90 per 1,000 successful requests. For a rank tracker doing a few thousand queries a month, that math usually beats paying for proxies plus CAPTCHA credits plus the engineer-hours to maintain selectors.

Which method should you choose?

Pick based on volume and how stable you need the output. Here is my decision rule.

• Learning or a one-off of under 50 queries: try Playwright locally, accept that some runs hit a consent or CAPTCHA wall, and do not build infrastructure for it.
• Any recurring job or commercial product: use a scraper API. Google’s layered defenses (JS gate, rotating selectors, IP limits, CAPTCHA) make DIY a maintenance treadmill, and a per-request API removes the proxy and CAPTCHA line items.
• You only need a handful of queries per day and can constrain to your own custom search engine: Google’s official Custom Search JSON API is free up to 100 queries/day, though it does not mirror the full public SERP.

The deciding factor is that a plain HTTP request returns 0 results, as my test showed, and closing that gap yourself means owning a browser farm, a proxy pool, and a CAPTCHA solver. For most readers that is the work an API already did.