How to Scrape Google News (Python Guide)

Google News aggregates headlines from thousands of publishers, which makes it the fastest single source for media monitoring, brand tracking, and dataset building. This guide builds a real Python scraper, tests it against the live service, and is honest about where the public web app blocks you. I ran the code on June 12, 2026 and pasted the real output below. The short version: the HTML page hides behind a consent wall and JavaScript, but Google’s own RSS feed hands you clean article data with no key required. For the fundamentals, see my web scraping guide and the Python walkthrough.

Can you scrape Google News, and is it legal?

You can pull public Google News data with a script, and the legal risk depends on the method and what you store. The technical reality first: the visible app at news.google.com/search runs on JavaScript and gates no-cookie clients behind a consent page, so a plain request gets nothing useful. The RSS feed, which I test below, is reachable with one GET. The legal side has two layers worth separating.

On US computer-access law, reading pages anyone can load without logging in sits on the safer side. In hiQ Labs v. LinkedIn, the Ninth Circuit held that scraping public data does not violate the Computer Fraud and Abuse Act, and the Supreme Court’s Van Buren decision narrowed the CFAA toward traditional hacking. Requesting a public RSS feed is not, by itself, a federal computer crime.

Two further constraints apply. Contract law is the live risk: Google’s Terms of Service restrict automated access, and hiQ won the CFAA question yet still paid a $500,000 judgment for breaching a user agreement. Copyright is the second: headlines and short factual snippets carry thin protection, but article bodies are owned by the publisher, so store links and metadata, not reproduced full text. Stay on public feeds, never script a logged-in Google account, and keep what you collect to headlines plus links. I am a tester, not a lawyer, so get advice before a commercial scrape. My is web scraping legal pillar covers the full picture.

What data can you extract from Google News?

The RSS feed carries the core article fields; the rendered web app carries more, but only if you can execute its JavaScript. Knowing which source holds what tells you when DIY is enough and when you need an API.

The takeaway: if you need headline, source, and date, the feed is enough. If you need snippets, images, or story clusters, you need the rendered page, which means a scraper API or a headless browser.

How does Google News block scrapers?

Google News stops naive scrapers with a consent wall, JavaScript rendering, and IP rate limits, in that order. When I requested the search page without cookies, the response was HTTP 200 but the body was the consent interstitial, not results. Here is the proof from my run on June 12, 2026:

import requests

UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " \
     "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"

r = requests.get(
    "https://news.google.com/search?q=openai&hl=en-US&gl=US&ceid=US:en",
    headers={"User-Agent": UA}, timeout=25,
)
print("final URL:", r.url)
print("consent wall:", "consent.google.com" in r.text.lower())
print("<article> tags:", r.text.lower().count("<article"))

final URL: https://consent.google.com/m?continue=https://news.google.com/search?q%3Dopenai...&gl=LT&m=0&pc=n&cm=2&hl=en-US&src=1
consent wall: True
<article> tags: 0

The request returned 200 and 633,965 bytes, but every byte was the consent page. Zero <article> tags means zero parseable results. Past the consent wall (which requires a SOCS or CONSENT cookie), the articles still load over JavaScript, so even with the cookie BeautifulSoup sees a shell. On top of that, a single IP firing rapid queries gets rate-limited. Three separate walls, and the plain-requests approach clears none of them.

Method 1: Scrape the Google News RSS feed with Python (DIY)

The DIY method that works is the RSS feed, parsed with requests and BeautifulSoup’s XML mode. The feed skips the consent wall and returns structured XML, so it is the right starting point for a free scraper. The endpoints:

The hl (language), gl (country), and ceid (country:language) parameters set localization. Here is the scraper:

import requests
from bs4 import BeautifulSoup

UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " \
     "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
URL = "https://news.google.com/rss/search?q=openai&hl=en-US&gl=US&ceid=US:en"

resp = requests.get(URL, headers={"User-Agent": UA}, timeout=25)
print("HTTP", resp.status_code, resp.headers["content-type"])

# Parse as XML, not HTML, so <source> and <pubDate> survive
soup = BeautifulSoup(resp.content, "xml")
items = soup.find_all("item")
print(f"Parsed {len(items)} articles\n")

for item in items[:5]:
    print(item.source.get_text() + " | " + item.pubDate.get_text())
    print("  " + item.title.get_text()[:72])

I ran this on June 12, 2026. Real output (headlines truncated to 72 chars):

HTTP 200 application/xml; charset=utf-8
Parsed 100 articles

WSJ | Thu, 11 Jun 2026 01:36:00 GMT
  Exclusive | OpenAI Considers Drastic Price Cuts, Anticipating War for Us
CNBC | Thu, 11 Jun 2026 03:24:00 GMT
  OpenAI mulls slashing prices as it competes with Anthropic for users: WS
Yahoo Finance | Fri, 12 Jun 2026 13:12:00 GMT
  A Price War Is Brewing Between OpenAI and Anthropic
The Guardian | Fri, 12 Jun 2026 09:30:00 GMT
  Behind the scenes at OpenAI HQ: the Stephen Collins cartoon - The Guardi
The Guardian | Thu, 11 Jun 2026 17:17:00 GMT
  Canadian mother sues OpenAI, alleging ChatGPT led her daughter to kill h

That is a working free scraper: one query, 100 real articles, title plus source plus date. Two limits to plan around. First, the <link> element is a news.google.com/rss/articles/... redirect, not the publisher’s URL, so resolving the canonical link means following the redirect in a browser session. Second, the feed stops near 100 items with no page parameter, and it carries no snippet, no thumbnail, and no top-stories cluster. Install the dependencies with pip install requests beautifulsoup4 lxml (the xml parser needs lxml). For more on this stack, see my BeautifulSoup guide and the requests walkthrough. If you need the full rendered page instead, Scrapy plus a headless browser is the heavier DIY route, with all the consent-cookie and rate-limit work falling on you.

Method 2: Scrape Google News with a scraper API

A scraper API is the right tool when you need the fields RSS omits or volume past a single IP. It renders the JavaScript, carries the consent cookie, rotates IPs, and returns parsed JSON, so you skip the three walls from the section above. ChocoData exposes a universal endpoint plus dedicated endpoints, and you point it at the Google News URL:

import requests

resp = requests.get(
    "https://api.chocodata.com/v1",
    params={
        "api_key": "YOUR_API_KEY",
        "url": "https://news.google.com/search?q=openai&hl=en-US&gl=US&ceid=US:en",
        "render": "true",       # execute the page JavaScript
        "country": "us",        # exit from a US IP
    },
    timeout=90,
)
print(resp.status_code)
data = resp.json()   # parsed articles: title, link, source, snippet, thumbnail

I did not run this call (it needs a paid key), so I am not pasting fake output. What an API buys you over the RSS route, by trade-off:

Decision rule: if headline, source, and date cover your use case, the free RSS scraper is enough and I would ship it. If you need snippets, images, resolved publisher links, or steady volume, the API earns its cost by handling rendering and IP rotation. For the wider anti-block toolkit (proxies, headers, retries), see scraping without getting blocked.

What are the limits of scraping Google News?

The honest limits: no official API, redirect links, a ~100-item cap, and a consent-walled HTML page. The RSS feed is undocumented and tolerated, so Google can change or pull it without notice, and link rewriting plus the item cap mean it is a discovery feed, not a full archive. The web app gives more fields but only to a client that runs its JavaScript and carries a consent cookie, which a plain script does not. Build on RSS for free headline monitoring, layer an API when you need rendered fields or scale, and store links and metadata rather than reproduced article bodies to stay on the right side of copyright.