How to Scrape Google Shopping (Python Guide)

Google Shopping is a price-comparison feed: one query returns the same product from many merchants with prices, ratings, and shipping side by side. That makes it the obvious target for price monitoring, MAP enforcement, and competitor research. It is also defended like the rest of Google Search. This guide builds a real Python scraper, runs it against the live Shopping page, and shows exactly where it breaks. I ran the code on June 12, 2026 and pasted the real output below, including the gate that stopped it. For the fundamentals first, see my web scraping guide and the Python scraping walkthrough.

Can you legally scrape Google Shopping?

Scraping public Shopping results sits in the same contested gray area as the rest of Google Search, and you should read two documents before you start. Google’s robots.txt is the clearest signal. I fetched it on June 12, 2026, and it disallows the exact paths a Shopping scraper targets:

User-agent: *
Disallow: /search
Disallow: /shopping/suppliers/search
Disallow: /shopping?
Disallow: /shopping/product/
Disallow: /shopping/search
Allow: /shopping?udm=28$

robots.txt is a crawling convention, not a statute, and ignoring it does not by itself create legal liability. The legal questions come from elsewhere:

The hiQ ruling leaned toward scrapers for openly accessible pages under the Computer Fraud and Abuse Act, and the Supreme Court’s Van Buren decision narrowed the CFAA toward traditional hacking. hiQ still paid a $500,000 judgment for breaching LinkedIn’s user agreement, so the contract layer is the live risk, and it is US-specific. My rule of thumb: scrape only public result pages, never log in, rate-limit hard, and keep personal data out of what you store. Prices and merchant names are not personal data, which keeps Shopping cleaner than people-search targets. For deeper coverage of blocking and ethics, see scraping without getting blocked. None of this is legal advice; if the data feeds a commercial product, talk to a lawyer.

What data can you extract from Google Shopping?

A Shopping results page packs more structured commerce data per query than almost any other Google surface. Each product tile carries the fields below, which is what makes the target attractive and what a scraper API should return as JSON.

Two things shape method choice. The product grid and the filters are rendered client-side by JavaScript, so a static HTML fetch sees none of them even when the status is 200. And prices vary by region and currency through the gl and hl parameters and the request IP, so output is not stable without geo control. Both points push any serious attempt toward a real browser or an API.

How does Google block Shopping scrapers?

Google blocks no-JS clients at the first request and escalates from there, the same stack that guards the main SERP. Here is what I observed and what comes next.

When I requested the Shopping page with requests and a desktop Chrome User-Agent, the response was HTTP 200 with the title “Google Search”, but the body was the enablejs bootstrap page, not the product grid. There were zero h3 titles and zero product blocks (sh-dgr__content, sh-dlr__list-result) to parse. The 200 is misleading: it is a loader, not the results.

If you push harder (more requests from one IP, obvious automation), Google escalates:

1. JavaScript gate (the enablejs bootstrap) - hits no-JS clients immediately, which is what I hit.
2. Client-side rendering - the product grid is built by JavaScript after load, so even a perfect HTML parse finds nothing in the initial document.
3. Rotating, obfuscated CSS class names - even with a real browser, selectors like sh-dgr__content change across regions and over time.
4. IP rate-limiting - bursts from a single datacenter IP trigger throttling.
5. The /sorry/ CAPTCHA page - a reCAPTCHA challenge once you look like a bot.
6. Region and currency variance - prices and availability differ by gl, hl, and the IP’s location.

The defenses are layered: JS execution, then rendering, then IP reputation, then a CAPTCHA wall. A scraper has to answer all of them.

Method 1: scrape Google Shopping with Python (the honest result)

A plain requests + BeautifulSoup script returns HTTP 200 but zero products, and here is the code plus the real output. I ran this on June 12, 2026 with Python 3.13.7, requests 2.34.2, and beautifulsoup4 4.14.3.

import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/124.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Sec-Fetch-Mode": "navigate",
}

url = "https://www.google.com/search?q=coffee+maker&tbm=shop&hl=en&gl=us"
r = requests.get(url, headers=HEADERS, timeout=25)
print("STATUS:", r.status_code)
print("BYTES:", len(r.content))

soup = BeautifulSoup(r.text, "html.parser")
title = soup.find("title")
print("TITLE:", title.get_text(strip=True) if title else None)
print("enablejs gate present:", "enablejs" in r.text)
print("h3 titles found:", len(soup.select("h3")))
print("product blocks found:",
      len(soup.select("div.sh-dgr__content, div.sh-dlr__list-result")))

The real output:

STATUS: 200
BYTES: 91435
TITLE: Google Search
enablejs gate present: True
h3 titles found: 0
product blocks found: 0

The 200 status fools beginners. Google returned the JavaScript bootstrap page, so there is no product grid in the HTML to select, which is why every count is 0. I reproduced this twice (the byte count drifted to 90,975 on the second run, normal for the bootstrap payload), and I did not find a cookie-less requests path that returns the product grid, so I am not pasting any parsed product fields. There were none.

The fix that actually renders the grid is a headless browser. Playwright executes the JavaScript, so the DOM contains real product nodes. Install with pip install playwright then playwright install chromium.

from playwright.sync_api import sync_playwright

def scrape_shopping(query: str):
    url = f"https://www.google.com/search?q={query}&tbm=shop&hl=en&gl=us"
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page(
            user_agent=(
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                "AppleWebKit/537.36 (KHTML, like Gecko) "
                "Chrome/124.0.0.0 Safari/537.36"
            ),
            locale="en-US",
        )
        page.goto(url, wait_until="domcontentloaded")
        # Consent screens appear in some regions; click through if present.
        try:
            page.click("button:has-text('Accept all')", timeout=3000)
        except Exception:
            pass

        products = []
        for tile in page.query_selector_all("div.sh-dgr__content"):
            title_el = tile.query_selector("h3, h4")
            price_el = tile.query_selector("span.a8Pemb, span.kHxwFf")
            if title_el:
                products.append({
                    "title": title_el.inner_text(),
                    "price": price_el.inner_text() if price_el else None,
                })
        browser.close()
        return products

if __name__ == "__main__":
    for i, row in enumerate(scrape_shopping("coffee+maker"), 1):
        print(i, row["price"], row["title"])

I am not pasting fake output for the Playwright version. On a fresh datacenter IP it frequently lands on a consent wall or the /sorry/ CAPTCHA, and the selectors (sh-dgr__content, a8Pemb) change across regions and over time, so results are inconsistent run to run. That is the honest state of DIY Google Shopping scraping. To make it production-grade you would add residential proxy rotation, per-region gl/hl handling, consent-screen logic, CAPTCHA solving, and selector maintenance every time Google reshuffles class names. For the browser-automation building blocks see the Scrapy guide and the BeautifulSoup guide.

DIY costs at scale add up fast:

Method 2: the official Content API for Shopping (your catalog only)

Google has an official Shopping API, and it is free, but it reads your own merchant data, not the public SERP. The Content API for Shopping (and its successor, the Merchant API) lets a registered merchant manage and read their product feed, inventory, and account in their own Merchant Center. It returns clean JSON and is fully sanctioned because you own the data.

The limit is the whole reason people scrape: the Content API does not return Shopping search results for queries you do not own. You cannot pull a competitor’s prices or a cross-merchant comparison for “coffee maker” through it. The request shape, for your own products, looks like this:

from googleapiclient.discovery import build
from google.oauth2 import service_account

# Service-account credentials from your Google Merchant Center / Cloud project
creds = service_account.Credentials.from_service_account_file(
    "service-account.json",
    scopes=["https://www.googleapis.com/auth/content"],
)
service = build("content", "v2.1", credentials=creds)

MERCHANT_ID = "1234567890"  # your own Merchant Center ID
products = service.products().list(merchantId=MERCHANT_ID).execute()
for p in products.get("resources", []):
    print(p["title"], p.get("price"))

I did not paste live output here because it needs a registered Merchant Center account and credentials, and I will not fake numbers. The endpoint and scope above come from Google’s developer docs. Use this route when the data you want is your own catalog. For everyone monitoring other merchants’ prices, it does not help, which is where Method 3 comes in.

Method 3: scrape Google Shopping with a scraper API

A scraper API turns the whole problem into one HTTP call that returns JSON, which is why most teams skip the browser stack for the public Shopping grid. The API runs the headless browser, rotates residential IPs, solves CAPTCHAs, and parses the page server-side. You send a query, you get structured product fields.

ChocoData is the service I use for this category. Per its own docs, it exposes a universal endpoint (“one endpoint, any site”) plus 453 dedicated endpoints, and the pattern is GET /api/v1/{site}/{resource}. The request shape:

import requests

API_KEY = "cd_live_YOUR_KEY"  # free tier: 1,000 requests/month, no card

resp = requests.get(
    "https://api.chocodata.com/api/v1/google/shopping",
    params={
        "q": "coffee maker",
        "gl": "us",            # country for pricing/currency
        "hl": "en",            # language
        "api_key": API_KEY,
    },
    timeout=60,
)
data = resp.json()  # JSON in, JSON out

for item in data.get("shopping_results", []):
    print(item.get("position"), item.get("price"),
          item.get("source"), item.get("title"))

If a Shopping-specific endpoint is not what you need, the same call style works against the universal endpoint by handing it the full Shopping URL with render=true and a country parameter, and parsing the returned HTML. Per ChocoData’s site, the service handles residential proxy rotation, CAPTCHAs, failed-request retries, and JavaScript rendering, bills only on successful (2xx) responses, and returns “clean structured JSON.” I have not independently benchmarked their success rate on Google Shopping, so treat the parsing claims as vendor-stated until you test on the free tier.

How the three methods compare on what actually matters:

ChocoData’s published pricing: a free tier of 1,000 requests/month with no card, then paid plans from $19/month (27,000 requests) up, plus pay-as-you-go top-ups at $0.90 per 1,000 successful requests. For a price monitor doing a few thousand queries a month, that math usually beats paying for proxies plus CAPTCHA credits plus the engineer-hours to maintain selectors.

Which method should you choose?

Pick based on whose data you need and at what volume. Here is my decision rule.

• You want your own products in Merchant Center: use the Content API for Shopping. It is free, structured, and fully sanctioned, and it never touches the anti-bot wall.
• You want competitor or cross-merchant prices, recurring or commercial: use a scraper API. A plain HTTP request returns 0 products, as my test showed, and closing that gap yourself means owning a browser farm, a proxy pool, and a CAPTCHA solver.
• Learning, or a one-off of under 50 queries on other merchants: try Playwright locally, accept that some runs hit a consent or CAPTCHA wall, and do not build infrastructure for it.

The deciding factor is the test result at the top: a bare requests call gets HTTP 200 and zero products because Google serves a JavaScript bootstrap, and the official API only reads your own catalog. For competitor pricing at any scale, a scraper API is the route that returns a real product grid instead of the gate I hit.

To go deeper on the building blocks, see the Python web scraping guide for the language fundamentals, BeautifulSoup for parsing, and Scrapy if you are crawling many queries and want a framework. The web scraping pillar ties the whole workflow together.