How to Scrape DuckDuckGo Search Results

DuckDuckGo is the privacy search engine people reach for when they want results without a Google account attached. That same privacy stance makes it awkward to scrape: there is no official search-results API, and the unofficial HTML endpoint throttles aggressively. I ran a Python scraper against it on 2026-06-12 to see exactly where the wall is. This guide covers whether you can scrape it, what data is on the page, how it blocks you, a DIY parser, and the scraper-API route that gets past the throttle.

For the broader playbook, see my web scraping guide. This article is the DuckDuckGo-specific layer on top.

Can you legally scrape DuckDuckGo organic results?

Scraping public DuckDuckGo result pages sits in the same legal space as scraping any public search engine: you are reading pages anyone can load without logging in. US courts have repeatedly held that scraping publicly accessible data does not violate the Computer Fraud and Abuse Act. In hiQ Labs v. LinkedIn, the Ninth Circuit ruled that scraping public profiles did not constitute “access without authorization” under the CFAA (9th Cir., 2022). The later Meta v. Bright Data case (N.D. Cal., 2024) reinforced that public data behind no login is fair game.

That said, three caveats apply to DuckDuckGo specifically:

• No login, no terms acceptance. The organic results page loads without an account, so you are not clicking through a terms-of-service gate the way you would on a logged-in product.
• The endpoint is unofficial. html.duckduckgo.com/html/ is a server-rendered version of the search UI, not a documented API. DuckDuckGo can change or throttle it at any time, and it does.
• Personal data and copyright still apply. Snippets can contain copyrighted text, and result pages can surface personal data. GDPR and copyright obligations attach to what you store and republish, regardless of how you fetched it.

I am describing what is technically and legally observable. This is not legal advice. If you are scraping at scale or commercially, get a lawyer to review your specific use case.

What data can you extract from a DuckDuckGo results page?

The organic results page gives you the standard search-result fields, plus a few DuckDuckGo-specific quirks. Here is what lives in the HTML when the page renders normally.

Two things DuckDuckGo does not give you:

• No result-count number. There is no “about 1,230,000 results” line, so you cannot estimate index size.
• No stable pagination cursor in plain HTML. The classic endpoint uses a s offset and a dc parameter passed back from the previous page, not a clean ?page=2. You have to carry forward form fields from the prior response.

One important framing point: DuckDuckGo sources its links from Bing’s index plus its own crawler. When you scrape DuckDuckGo organic results, you are reading a Bing-derived ranking with DuckDuckGo’s own filtering on top. If your goal is rank tracking, decide whether DuckDuckGo’s specific ordering is what you need or whether Bing is the real target.

How does DuckDuckGo block scrapers?

DuckDuckGo blocks with a JavaScript anomaly challenge and an HTTP 202 status, and it triggers fast. This is the part I tested, and the result was clear.

I wrote a plain requests + BeautifulSoup scraper that POSTs a query to html.duckduckgo.com/html/ and parses div.result__body. A first manual curl from a clean IP returned HTTP 200. Every subsequent request from my script returned HTTP 202 with zero parseable results. Inspecting the 202 body showed why:

GET  -> HTTP 202 | bytes  14253 | results 0 | title='DuckDuckGo' | flags=['anomaly']
POST -> HTTP 202 | bytes  14247 | results 0 | title='DuckDuckGo' | flags=['anomaly']

The 202 page is not the results page. It contains a hidden form that loads DuckDuckGo’s anomaly script with rotating, signed tokens:

<form id="img-form" action="//duckduckgo.com/anomaly.js?sv=html&cc=sre
  &ti=1781279777&gk=d4cd0dabcf4caa22ad92fab40844c786
  &p=19a35de270e182a720528e91465f7484-9d5e5407b2f04d4f295d2fc1814c6ddd-..."

That anomaly.js request, with the gk and p tokens, is a browser-side challenge. A real browser runs the script, the challenge resolves, and results load. requests does not execute JavaScript, so it stays stuck on the 202 page forever. Retrying immediately does not help; I retried six times and got 202 every time once the IP was flagged.

To be explicit about my own run: I could not parse a single organic result with plain HTTP from my IP on 2026-06-12 because of this 202 anomaly challenge. So I am not pasting fake result rows. The honest takeaway: DIY plain-HTTP scraping works for a query or two from a fresh IP, then DuckDuckGo throttles you to 202.

Here is how the blocking behavior breaks down by trigger.

The pattern is the same on lite.duckduckgo.com and the main duckduckgo.com/html/ host: simpler HTML, same anomaly throttle.

Method 1: DIY scraper with Python (requests + BeautifulSoup)

For a few queries from a clean residential IP, plain Python works. Here is the scraper I ran. It POSTs to the HTML endpoint and parses the organic blocks.

import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}

def scrape_ddg(query, max_results=10):
    url = "https://html.duckduckgo.com/html/"
    resp = requests.post(url, data={"q": query}, headers=HEADERS, timeout=20)

    # DuckDuckGo returns 202 with a JS anomaly challenge when it throttles you.
    if resp.status_code == 202 or "anomaly" in resp.text.lower():
        raise RuntimeError(
            f"Blocked: HTTP {resp.status_code}, anomaly challenge served. "
            "Rotate IP or use a scraper API."
        )
    resp.raise_for_status()

    soup = BeautifulSoup(resp.text, "html.parser")
    results = []
    for block in soup.select("div.result__body")[:max_results]:
        a = block.select_one("a.result__a")
        if not a:
            continue
        snippet_el = block.select_one("a.result__snippet")
        results.append({
            "title": a.get_text(strip=True),
            "url": a.get("href"),
            "snippet": snippet_el.get_text(strip=True) if snippet_el else "",
        })
    return results

if __name__ == "__main__":
    for r in scrape_ddg("web scraping python", max_results=5):
        print(r["title"])
        print(" ", r["url"])
        print(" ", r["snippet"][:100])
        print()

When I ran this on 2026-06-12, it raised the RuntimeError because the endpoint returned 202. That is the realistic outcome from a datacenter or repeat-hit IP. The parsing logic itself is correct: when the page does render (fresh residential IP, first request), div.result__body is the right selector and the title/url/snippet fields populate as shown in the data table above.

Three things make the difference between this working and 202:

• Rotate residential IPs. One request per IP per query is the safe envelope. Datacenter IPs get 202 almost immediately.
• Throttle hard. Several seconds between requests, not milliseconds. The anomaly system keys heavily on request rate.
• You still cannot solve the JS challenge with requests. Once you see 202, no header tweak fixes it from plain HTTP. You either switch IP before the flag or move to a browser engine.

If you want the headless-browser route, a tool like Playwright can execute anomaly.js and let the page resolve. That works, but it is heavy: a full Chromium per query, plus you still need IP rotation to avoid the challenge re-firing. For more on that trade-off, see scraping without getting blocked. The general BeautifulSoup and Python mechanics are in my BeautifulSoup guide and Python scraping guide.

Method 2: Scrape DuckDuckGo with a scraper API

A scraper API solves the 202 problem by running a real browser with rotating residential IPs on its side, so you get parseable HTML back instead of an anomaly page. This is the route I reach for when I need DuckDuckGo results at any volume, because it removes both the JavaScript challenge and the IP-rotation chore.

ChocoData is the scraper API I use for search-engine pages. It exposes a universal endpoint plus 453 dedicated endpoints; you send the target URL, it handles browser rendering and proxy rotation, and returns the HTML. You then parse with the same BeautifulSoup selectors from Method 1.

import requests
from bs4 import BeautifulSoup

API_KEY = "YOUR_CHOCODATA_KEY"

def scrape_ddg_via_api(query, max_results=10):
    target = "https://html.duckduckgo.com/html/?q=" + requests.utils.quote(query)
    resp = requests.get(
        "https://api.chocodata.com/v1",
        params={
            "api_key": API_KEY,
            "url": target,
            "render_js": "true",   # executes anomaly.js so results render
        },
        timeout=90,
    )
    resp.raise_for_status()

    soup = BeautifulSoup(resp.text, "html.parser")
    results = []
    for block in soup.select("div.result__body")[:max_results]:
        a = block.select_one("a.result__a")
        if not a:
            continue
        snippet_el = block.select_one("a.result__snippet")
        results.append({
            "title": a.get_text(strip=True),
            "url": a.get("href"),
            "snippet": snippet_el.get_text(strip=True) if snippet_el else "",
        })
    return results

The selectors are identical to the DIY version. The only change is that the request goes through the API, which returns the rendered results page instead of the 202 challenge. Confirm the exact parameter names (render_js, country/geo flags, the endpoint host) against the current ChocoData docs before you ship, since API params change more often than HTML selectors.

Here is the honest decision table for picking a method.

When is scraping DuckDuckGo the wrong call?

Scraping DuckDuckGo is the wrong call when you actually need the underlying Bing ranking or a structured answer box, because there are cleaner sources for both. Three cases where I would not scrape the DuckDuckGo HTML page:

• You need the source ranking. DuckDuckGo derives organic links from Bing. If Bing is your real target, scrape Bing directly and skip DuckDuckGo’s filtering layer.
• You need zero-click facts. DuckDuckGo’s Instant Answer API (api.duckduckgo.com/?q=...&format=json) returns the info-box content with no anomaly throttle. Use it for definitions, entities, and disambiguation, never for the organic list.
• You need guaranteed volume on a deadline. The 202 throttle makes DIY unpredictable. If a job has to finish reliably, the scraper-API route is the realistic path.

For everything else, the workflow is simple: parse div.result__body with the selectors above, expect 202 on plain HTTP after the first request, and add rendering plus IP rotation (a headless browser or a scraper API) once you scale past a handful of queries. If you want a heavier framework for crawling many result pages, my Scrapy guide covers structuring that as a spider.