How to Scrape News Sites (Python Guide)

News sites are the most uneven scraping target I test. Some hand you clean server-rendered HTML with structured metadata baked in; others return a 401 before you read a single headline. This guide builds a real Python scraper for news article pages, tests it against five live outlets, and is honest about which ones block and why. I ran every snippet on June 12, 2026 and pasted the real output below. For the fundamentals first, see my web scraping guide and the Python walkthrough. If your target is the Google News aggregator rather than a publisher site, I cover that separately in how to scrape Google News.

Can you scrape news sites, and is it legal?

You can scrape public news pages with a script, but a 200 response is a technical signal, not legal permission, and the two often disagree. The technical question is whether the server hands you the HTML; the legal question is whether the publisher’s terms and copyright let you keep and use it. They are separate, and on news sites they frequently point in opposite directions.

On US computer-access law, reading pages anyone can load without logging in sits on the safer side. In hiQ Labs v. LinkedIn, the Ninth Circuit held that scraping public data does not violate the Computer Fraud and Abuse Act, and the Supreme Court’s Van Buren decision narrowed the CFAA toward traditional hacking. Requesting a public article URL is not, by itself, a federal computer crime.

Two constraints do the real work here. Contract law is the live risk: terms of service restrict automated access, and hiQ won the CFAA question yet still paid a $500,000 judgment for breaching a user agreement. Copyright is the second and bites harder on news than almost anywhere: an article body is an original work owned by the publisher. Headlines and short factual snippets carry thin protection, so the defensible pattern is to store links, headlines, dates, and a brief excerpt, never reproduced full text. I am a tester, not a lawyer, so get advice before a commercial scrape. My is web scraping legal pillar covers this in depth.

What data can you extract from a news article page?

A modern news article page carries a rich, predictable set of fields, and most of them live in a structured-data block rather than scattered through the HTML. Knowing the field map tells you what to target and which extraction method is least likely to break.

The pattern that survives redesigns: pull metadata from JSON-LD and Open Graph tags first, and treat CSS selectors on the body as the fragile part you expect to fix later.

Which news sites actually let you in? (live test)

I sent one plain GET request, with a normal browser user agent, to five major outlets on June 12, 2026. The results split sharply. Here is the exact code:

import requests

headers = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
    )
}
sites = [
    "https://www.bbc.com/news",
    "https://www.reuters.com/",
    "https://apnews.com/",
    "https://techcrunch.com/",
    "https://www.theguardian.com/international",
]
for url in sites:
    try:
        r = requests.get(url, headers=headers, timeout=20)
        print(url, "->", r.status_code, "len", len(r.text))
    except Exception as e:
        print(url, "-> ERR", type(e).__name__)

The real output:

https://www.bbc.com/news -> 200 len 404729
https://www.reuters.com/ -> 401 len 771
https://apnews.com/ -> 200 len 2140542
https://techcrunch.com/ -> 200 len 447447
https://www.theguardian.com/international -> 200 len 1494110

Four of five returned a full page. Reuters returned 401 Unauthorized with a 771-byte body, which is a bot-mitigation block rather than real content. Two takeaways. First, you cannot assume any given news site is scrapable; test it before you build. Second, the BBC’s 200 is misleading on the legal axis, which the next section covers. I built the rest of this guide on the Guardian because it server-renders clean HTML and, unusually, also offers an official API.

How do news sites block scrapers?

News sites lean on the same defenses as any high-traffic property, plus a few of their own, and the block can arrive at any layer from the TCP handshake to the rendered DOM. Recognizing which one you hit tells you whether to tweak headers or reach for a different tool.

The honest read: header tweaks fix consent walls and light rate limits. They do not fix a Fastly or Akamai 401 like Reuters, and they do not bypass a paywall. For those, you either license a feed or route through infrastructure that solves the challenge for you.

Method 1: scrape a news article with Python (requests + BeautifulSoup)

For a server-rendered site like the Guardian, the most durable method is fetching the page and reading its JSON-LD NewsArticle block, falling back to the DOM only for body text. Here is the scraper I ran against a live Guardian article:

import requests, json
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
           "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"}

def get_article(url):
    r = requests.get(url, headers=HEADERS, timeout=20)
    r.raise_for_status()
    soup = BeautifulSoup(r.text, "html.parser")

    # 1. Pull the JSON-LD NewsArticle block - the structured-data source.
    ld = {}
    for s in soup.find_all("script", type="application/ld+json"):
        try:
            j = json.loads(s.string or "{}")
        except json.JSONDecodeError:
            continue
        for it in (j if isinstance(j, list) else [j]):
            if isinstance(it, dict) and it.get("@type") in (
                    "NewsArticle", "Article", "ReportageNewsArticle"):
                ld = it

    authors = ld.get("author", [])
    if isinstance(authors, dict):
        authors = [authors]
    author_names = [a.get("name") for a in authors if isinstance(a, dict)]

    # 2. Body paragraphs come from the article DOM.
    paras = [p.get_text(strip=True)
             for p in soup.select("div[data-gu-name='body'] p")]
    body = "\n\n".join(paras)

    return {
        "url": url,
        "headline": ld.get("headline"),
        "published": ld.get("datePublished"),
        "modified": ld.get("dateModified"),
        "authors": author_names,
        "word_count": len(body.split()),
        "first_para": paras[0] if paras else None,
    }

art = get_article("https://www.theguardian.com/world/2026/jun/12/"
                  "thailand-princess-bajrakitiyabha-dies-aged-47")
for k, v in art.items():
    print(f"{k:11}: {v}")

That returned clean structured fields from the live page:

url        : https://www.theguardian.com/world/2026/jun/12/thailand-princess-bajrakitiyabha-dies-aged-47
headline   : Thailand's Princess Bajrakitiyabha dies aged 47 after years in a coma
published  : 2026-06-12T01:04:59.000Z
modified   : 2026-06-12T01:29:17.000Z
authors    : ['Rebecca Ratcliffe']
word_count : 440
first_para : The eldest child of Thailand's King Maha Vajiralongkorn has died aged 47...

Two things matter here. The JSON-LD gave me the headline, both ISO timestamps, and the author with zero selector guesswork, because publishers maintain that block for Google. The body, by contrast, came from div[data-gu-name='body'] p, a Guardian-specific selector that will break the day they restructure their templates. That division of labor (metadata from structured data, body from the DOM) is the pattern to copy on any news site. For the selector mechanics, see my guides on BeautifulSoup and XPath and CSS selectors.

To collect many articles, scrape a section front page for links first, then loop get_article over them. I pulled the homepage links the same way:

soup = BeautifulSoup(requests.get(
    "https://www.theguardian.com/international", headers=HEADERS).text, "html.parser")
links = {a["href"] for a in soup.select("a[data-link-name]")
         if "/2026/" in a.get("href", "")
         and len(a.get_text(strip=True)) > 30}
print(len(links), "unique article links")

21 unique article links

Always read the site’s robots.txt before you loop, add a delay between requests, and stay off disallowed paths. Mind that the Guardian’s robots.txt disallows /search and /discussion/*, so a link harvester should skip those.

Method 2: use a news scraper API (handles blocks and JS)

When the site blocks you like Reuters, renders headlines in JavaScript, or you need to pull many outlets at scale without maintaining proxies, a news scraper api is the pragmatic route. You send one HTTP request with the target URL; the service rotates IPs, solves the bot challenge, runs a headless browser when needed, and returns the HTML or parsed JSON. ChocoData works this way, exposing a universal endpoint plus 453 dedicated endpoints, with a render flag for JavaScript-heavy pages.

The call is a single GET. You hand it the URL, it hands back the page your own request could not reach:

import requests, json
from bs4 import BeautifulSoup

API_KEY = "YOUR_CHOCODATA_KEY"

resp = requests.get(
    "https://api.chocodata.com/v1/universal",
    params={
        "api_key": API_KEY,
        "url": "https://www.reuters.com/world/",
        "render": "true",   # execute JavaScript
    },
    timeout=90,
)
print("STATUS:", resp.status_code)

# The API returns the fully rendered HTML; parse it exactly as before.
soup = BeautifulSoup(resp.text, "html.parser")
print("headlines found:", len(soup.select("a[data-testid='Heading']")))

I have not pasted output for this snippet because it needs a paid key, and my rule is to show only output I actually ran. The mechanics are what matter: the same BeautifulSoup parsing you wrote for Method 1 runs unchanged on the returned HTML, so a scraper API slots into existing code as a drop-in fetch layer. For deeper background on why this beats a homegrown proxy pool, read scraping without getting blocked.

Here is when each method is the right call:

What are the limits of scraping news data?

Scraping gets you headlines, metadata, and snippets reliably; it does not get you a clean license to the full text, and that is the real ceiling. Three limits to plan around.

First, copyright caps what you can store and reuse. Bodies belong to the publisher, so build for monitoring (link, headline, date, snippet) and link back, rather than building an archive of reproduced articles. Second, selectors rot. Metadata in JSON-LD is stable because Google consumes it, but the body selectors I used for the Guardian are outlet-specific and will break on a redesign; budget for maintenance. Third, blocks and rate limits scale with you. One article a minute from one outlet is easy; thousands of articles an hour across many outlets means proxies, retries, and challenge-solving, which is the work a scraper API absorbs.

Where an official API exists, prefer it. The Guardian’s Open Platform Developer key is free and gives you 1 call per second, 500 calls per day, full article text, and access to over 1,900,000 pieces of content under documented terms. That is a contract-backed feed, which beats scraping on both reliability and legal clarity. For everything else, the honest answer is the one this guide demonstrates: test the target, parse structured data first, respect the terms, and reach for a scraper API the moment the site fights back.