How to Scrape Baidu Search Results

I tried to scrape Baidu the simple way and pasted both real outcomes below. Short version: the first request from a clean IP returns HTTP 200 with server-rendered HTML, and I parsed 8 organic results from it. Push a little harder and Baidu flags the IP, then every request comes back HTTP 200 with a security-verification page and zero results. This guide covers whether you can scrape Baidu, what the results page holds, the exact block you will hit, a working DIY parser with real output, and the scraper-API path that returns clean JSON. For fundamentals, see my web scraping guide and the Python scraping walkthrough.

Can you legally scrape Baidu search results?

Scraping public SERPs is a contested gray area, and for Baidu you should read its robots.txt before writing any code. I fetched https://www.baidu.com/robots.txt on June 12, 2026 (HTTP 200, 2,814 bytes). It names a long list of search-engine bots and disallows the search and redirect paths for each one:

User-agent: Baiduspider
Disallow: /baidu
Disallow: /s?
Disallow: /ulink?
Disallow: /link?
Disallow: /home/news/data/
Disallow: /bh
...
User-agent: *
Disallow: /

Two lines matter. Disallow: /s? is the search results path you would scrape, and the file ends with User-agent: * / Disallow: /, a blanket request for every unnamed crawler to fetch nothing. robots.txt is a crawling convention, not a statute, so it does not by itself create legal liability. The weight comes from elsewhere:

The hiQ ruling leaned toward scrapers for openly accessible US pages, but it is US-specific and did not settle breach-of-contract claims. Baidu is a Chinese company governed by China’s PIPL and Data Security Law, so the calculus differs from a US target. My rule of thumb: scrape only public result pages, never log in, rate-limit hard, and keep personal data out of what you store. For deeper coverage of blocking and ethics, see scraping without getting blocked. None of this is legal advice; if the data feeds a commercial product, talk to a lawyer.

What data can you extract from a Baidu SERP?

A Baidu results page is a stack of distinct blocks, and each carries different fields. Knowing the layout tells you what to target and what an API should return as JSON.

Two things shape your method. First, organic links are wrapped in /link?url=... redirect URLs, so the displayed result hides its true destination until you resolve the hop. Second, the Aladdin rich cards are populated by JavaScript, so a static fetch misses them even when the base markup loads. The plain organic list, though, is server-rendered, which is why a first cold request can be parsed without a browser.

How does Baidu block scrapers?

Baidu serves real HTML to the first cold request, then flags the IP and swaps in a security-verification page that still returns HTTP 200. Here is what I observed and what comes after it.

My first request to https://www.baidu.com/s?wd=python+web+scraping with a desktop Chrome user agent and a zh-CN Accept-Language returned HTTP 200, about 1.2 MB of HTML, the title python web scraping_百度搜索, and 8 parseable organic results under #content_left. That is genuine server-rendered output. Within a handful of follow-up requests from the same IP, the response changed: still HTTP 200, but only 1,438 bytes, with the title 百度安全验证 (“Baidu Security Verification”) and the body message 网络不给力，请稍后重试 (“the network is busy, please retry later”). Zero results. The challenge page loads a JavaScript fingerprint script from wappass/bcebos and expects the browser to solve it.

So the defenses escalate in order:

1. Cold request - the organic SERP is server-rendered, so one clean fetch parses fine.
2. IP and behavior reputation - bursts from one IP, no cookies, and no JS execution flag you fast.
3. Security-verification page - Baidu returns a 1,438-byte 百度安全验证 page under a deceptive HTTP 200 instead of results.
4. /link?url= redirect throttling - resolving each result’s true URL is a second rate-limited hop.
5. Geo and market variance - results, ads, and the challenge threshold change with the IP’s location; a non-Chinese datacenter IP trips the wall sooner.

The trap is the status code. A scraper that only checks r.status_code == 200 will happily store thousands of empty verification pages. You have to inspect the body.

Method 1: DIY scraping with Python (the honest result)

A plain requests + BeautifulSoup script parses the first cold request, then gets soft-blocked, and here is the code plus both real outputs. I ran this on June 12, 2026 with Python 3.13.7, requests 2.34.2, and beautifulsoup4 4.14.3.

import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
}


def scrape_baidu(query, page=0):
    """Return organic results from one Baidu SERP page. page is zero-based; pn = page * 10."""
    params = {"wd": query, "pn": page * 10}
    resp = requests.get(
        "https://www.baidu.com/s", params=params, headers=HEADERS, timeout=20
    )
    resp.raise_for_status()
    resp.encoding = "utf-8"

    # Detect the security-verification page (HTTP 200 but no results).
    if "百度安全验证" in resp.text:
        raise RuntimeError("Blocked: Baidu served a security-verification page")

    soup = BeautifulSoup(resp.text, "html.parser")
    out = []
    for box in soup.select("#content_left div.result, #content_left div.c-container"):
        link = box.select_one("h3 a")
        if not link or not link.get_text(strip=True):
            continue
        out.append({
            "rank": len(out) + 1,
            "title": link.get_text(strip=True),
            "redirect_url": link.get("href", ""),  # /link?url=... wrapper
        })
    return out


if __name__ == "__main__":
    rows = scrape_baidu("python web scraping")
    print(f"Parsed {len(rows)} organic results\n")
    for r in rows:
        print(f"#{r['rank']:>2}  {r['title'][:60]}")
        print(f"     {r['redirect_url'][:62]}")

The real output from the first cold request:

Parsed 8 organic results

# 1  网页抓取(网页搜集方法) - 百度百科
     http://www.baidu.com/link?url=wYJMh5LvPF2Y4wb226S9DNLIzAZiEFR5
# 2  PythonWebScrapingSolutions for Automated Data
     http://www.baidu.com/link?url=DGVqB1AKMZsLM4of70t2VthhZok4Ie8s
# 3  Pythonwebscrapingtutorial
     http://www.baidu.com/link?url=4sJmiSilsF-xKcxox6Gs4JaPYQOxF6ni
# 4  WebScrapingPython
     http://www.baidu.com/link?url=KbgEhHtOUeW4rx6oFKlqtotvXCW5uyl1
# 5  Python使用Scrapling进行网页采集的用法详解_python_脚本之家
     http://www.baidu.com/link?url=wYJMh5LvPF2Y4wb226S9DL1m9j9CwJx2
# 6  web-scraping-python· GitHub Topics · GitHub
     http://www.baidu.com/link?url=DzFEBtXZgNIzP3i56sR9vKhjEabHgqPF
# 7  Python+BeautifulSoupWebScraping工程化实战指南
     http://www.baidu.com/link?url=KbgEhHtOUeW4rx6oFKlqtbD8Y6FkAD9A
# 8  PythonWebScraping: Step-By-Step Tutorial
     http://www.baidu.com/link?url=lqVHFf3OZa4zEMcFXKUfHkdaIyx8Gfm8

That is genuine server-rendered data: titles, ranks, and the /link?url= redirect wrappers. The catch is durability. After a few requests from the same IP, the identical script hit the wall. I added a one-line diagnostic to dump status, title, and length, and the real output was:

STATUS: 200
page title: '百度安全验证'
response length: 1438
security_verification_page: True
result containers: 0
organic links parsed: 0

HTTP 200, correct-looking from a status check, 1,438 bytes, zero results. The if "百度安全验证" in resp.text guard in the function above is what turns that silent failure into a raised exception so you do not store empty pages.

Two more limits before you scale this. The redirect_url values are /link?url= wrappers, so to get the real destination you must send each one and follow the redirect, which doubles your request count and your block risk. And once the IP is flagged, it stayed flagged across my later runs, so a time.sleep() between requests does not reliably clear it. To make this production-grade you would add: Chinese or residential proxy rotation, a real browser to solve the verification fingerprint, redirect resolution, and selector upkeep when Baidu reshuffles its markup. For browser-automation fundamentals see the Scrapy guide and BeautifulSoup guide.

DIY costs at scale add up fast:

Method 2: Scraping Baidu with a scraper API

A scraper API turns the whole problem into one HTTP call that returns structured data, which is why most teams skip the proxy and browser stack. The API runs a headless browser, rotates residential IPs (including Chinese exits), clears the verification page, and hands back parsed fields. You send a URL, you get data.

ChocoData (chocodata.com) is one example I checked against its own docs. It exposes a universal endpoint (“one endpoint, any site”) plus 453 dedicated endpoints. For Baidu, the universal endpoint with JavaScript rendering is the straightforward route. The documented request shape:

import requests

API_KEY = "YOUR_API_KEY"  # free tier: 1,000 requests/month, no card

resp = requests.get(
    "https://chocodata.com/api/v1/universal",
    params={
        "url": "https://www.baidu.com/s?wd=python+web+scraping",
        "render": "true",            # run the headless browser
        "country": "cn",             # request a Chinese exit IP
        "api_key": API_KEY,
    },
    timeout=60,
)
data = resp.json()  # rendered HTML or parsed fields, depending on options

The endpoint pattern is GET /api/v1/universal?url=... for any page, and GET /api/v1/{site}/{resource}?api_key=... for dedicated endpoints. Per ChocoData’s site, the service handles residential proxy rotation, CAPTCHAs and challenge pages, failed-request retries, and optional JavaScript rendering. I have not independently benchmarked their Baidu success rate, so treat their parsing claims as vendor-stated until you test on the free tier.

How the two methods compare on the things that actually matter:

ChocoData’s published pricing: a free tier of 1,000 requests/month with no card, then paid plans from $19/month (27,000 requests) up, and pay-as-you-go top-ups at $0.90 per 1,000 successful requests. For a Baidu rank tracker doing a few thousand queries a month, that math usually beats paying for Chinese proxies plus CAPTCHA credits plus the engineer-hours to maintain selectors and redirect resolution.

Which method should you choose?

Pick based on volume and how stable you need the output. Here is my decision rule.

• Learning or a one-off of under 20 queries: run the DIY parser above, accept that your IP gets flagged fast, and do not build infrastructure for it.
• Any recurring job or commercial product: use a scraper API. Baidu’s IP reputation system, the deceptive-200 verification page, and the /link?url= redirects make DIY a maintenance treadmill, and a per-request API removes the proxy, CAPTCHA, and redirect-resolution line items.
• You only need a handful of fields from the first page occasionally: the cold-request parser is enough, as long as you keep the 百度安全验证 guard so you never store empty pages.

The deciding factor is durability. A single clean request parses real Baidu results, as my test showed, but the IP gets flagged within a few requests and the page silently degrades to a 200 challenge. Closing that gap yourself means owning a Chinese proxy pool, a browser that solves Baidu’s fingerprint, and a redirect resolver. For most readers that is work an API has already done.