How to Scrape Bing Search Results

I tried to scrape Bing the simple way and pasted the real result below. Short version: a plain Python requests call returns HTTP 200 with the right page title, but zero results, because Bing ships a JavaScript shell to clients that do not run JS. This guide covers whether you can scrape Bing, what the results page holds, the exact failure you will hit, a Playwright attempt, and the scraper-API path that returns clean JSON. For the fundamentals, see my web scraping guide and the Python scraping walkthrough.

Can you legally scrape Bing search results?

Scraping public SERPs is a contested gray area, and for Bing you should read three documents before writing any code. Bing’s robots.txt is the clearest signal. I fetched it on June 12, 2026 and it disallows the search path twice, for both casings:

User-agent: *
...
Disallow: /results
Disallow: /search
Disallow: /Search
Disallow: /settings

robots.txt is a crawling convention, not a statute, so it does not by itself create legal liability. The legal weight comes from elsewhere:

The hiQ ruling concerned public data and the Computer Fraud and Abuse Act, and it leaned toward scrapers for openly accessible pages. It did not settle breach-of-contract claims, and it is US-specific. My rule of thumb: scrape only public result pages, never log in, rate-limit hard, and keep personal data out of what you store. For deeper coverage of blocking and ethics, see scraping without getting blocked. None of this is legal advice; if the data feeds a commercial product, talk to a lawyer.

What data can you extract from a Bing SERP?

A Bing results page is a stack of distinct blocks, and each block carries different fields. Knowing the layout tells you what to target and what an API should return as JSON.

Two of these decide your method. People Also Ask and the image/video packs are populated by JavaScript, so a static HTML fetch would miss them even if Bing served you the base markup. That alone pushes any serious DIY attempt toward a real browser.

How does Bing block scrapers?

Bing blocks no-JS clients at the response body, then escalates on IP behavior. Here is what I observed and what comes after it.

When I requested the search page with requests and a desktop Chrome user agent, the response was HTTP 200 with the correct title (web scraping python - Search), but the body was mostly JavaScript: about 44 KB of script in a 71 KB page, with only 6 anchors and no <li> elements. The organic result markup is not in the HTML. The 200 is misleading: it is a bootstrap shell, not the SERP. Adding a warm-up request to collect Bing’s cookies (MUID, SRCHUID, _EDGE_S, and others) changed nothing; the result HTML still had zero organic blocks.

If you push harder from one IP, Bing escalates:

1. JavaScript shell - the result body is not server-rendered for no-JS clients, so a plain fetch sees 0 results.
2. Cookie and session checks - Bing sets MUID and SRCH* cookies; repeat requests without sane session behavior look automated.
3. IP rate-limiting - bursts from a single datacenter IP get throttled.
4. CAPTCHA challenge - sustained bot-like traffic triggers a verification wall.
5. Region and market variance - results change by the mkt, cc, and setlang parameters and by the IP’s location, so output is unstable without geo control.

So the defenses are layered: a JS-rendered body first, then session and IP reputation, then a CAPTCHA. A scraper has to answer all three.

Method 1: DIY scraping with Python (the honest result)

A plain requests + BeautifulSoup script returns HTTP 200 but zero results, and here is the code plus the real output. I ran this on June 12, 2026 with Python 3.13.7 and requests 2.34.2.

import requests
from bs4 import BeautifulSoup

headers = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/124.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}

url = "https://www.bing.com/search?q=web+scraping+python&count=10"
r = requests.get(url, headers=headers, timeout=30)
print("STATUS:", r.status_code)

soup = BeautifulSoup(r.text, "html.parser")
print("page title:", soup.title.string if soup.title else None)
print("li.b_algo results found:", len(soup.select("li.b_algo")))
print("h2 result links found:", len(soup.select("h2 a")))
print("total <li> in DOM:", len(soup.find_all("li")))

The real output:

STATUS: 200
page title: web scraping python - Search
li.b_algo results found: 0
h2 result links found: 0
total <li> in DOM: 0

The 200 status and the correct page title fool beginners. The page is the JavaScript shell, so there is nothing to parse: no li.b_algo blocks, no <h2> links, no <li> at all. The fix that actually renders results is a headless browser. Playwright executes the JavaScript, so the DOM contains real result nodes. Install with pip install playwright then playwright install chromium.

from playwright.sync_api import sync_playwright

def scrape_bing(query: str):
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page(
            user_agent=(
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                "AppleWebKit/537.36 (KHTML, like Gecko) "
                "Chrome/124.0.0.0 Safari/537.36"
            ),
            locale="en-US",
        )
        page.goto(
            f"https://www.bing.com/search?q={query}&count=10",
            wait_until="domcontentloaded",
        )
        page.wait_for_selector("li.b_algo", timeout=8000)

        results = []
        for block in page.query_selector_all("li.b_algo"):
            title_el = block.query_selector("h2 a")
            snippet_el = block.query_selector(".b_caption p, .b_lineclamp2")
            if title_el:
                results.append({
                    "title": title_el.inner_text(),
                    "url": title_el.get_attribute("href"),
                    "snippet": snippet_el.inner_text() if snippet_el else "",
                })
        browser.close()
        return results

if __name__ == "__main__":
    for i, row in enumerate(scrape_bing("web+scraping+python"), 1):
        print(i, row["title"], "->", row["url"])

I am not pasting output for the Playwright version, because I did not run it to a clean, repeatable result. On a fresh datacenter IP it can land on a CAPTCHA or a sparse market, and the wait_for_selector call times out when Bing serves a layout where organic blocks render late or under a different class. That is the honest state of DIY Bing scraping. To make it production-grade you would add: residential proxy rotation, per-market mkt/setlang handling, CAPTCHA solving, and selector maintenance whenever Bing reshuffles its markup. For browser-automation fundamentals see the Scrapy guide and BeautifulSoup guide.

DIY costs at scale add up fast:

Method 2: Scraping Bing with a scraper API

A scraper API turns the whole problem into one HTTP call that returns JSON, which is why most teams skip the browser stack. The API runs the headless browser, rotates residential IPs, clears CAPTCHAs, and hands back parsed fields. You send a URL, you get structured data.

ChocoData (chocodata.com) is one example I checked against its own docs. It exposes a universal endpoint (“one endpoint, any site”) across 235 sites plus 453 dedicated endpoints. For Bing, the universal endpoint with JavaScript rendering is the straightforward route. The documented request shape:

import requests

API_KEY = "YOUR_API_KEY"  # free tier: 1,000 requests/month, no card

resp = requests.get(
    "https://chocodata.com/api/v1/universal",
    params={
        "url": "https://www.bing.com/search?q=coffee+maker&count=10",
        "render": "true",          # run the headless browser
        "api_key": API_KEY,
    },
    timeout=60,
)
data = resp.json()  # rendered HTML or parsed fields, depending on options

The endpoint pattern is GET /api/v1/{site}/{resource}?api_key=... for dedicated endpoints, and GET /api/v1/universal?url=... for any page. Per ChocoData’s site, the service handles residential proxy rotation, CAPTCHAs, failed-request retries, and optional JavaScript rendering. I have not independently benchmarked their Bing success rate, so treat their parsing claims as vendor-stated until you test on the free tier.

How the two methods compare on the things that actually matter:

ChocoData’s published pricing: a free tier of 1,000 requests/month with no card, then paid plans from $19/month (27,000 requests) up, and pay-as-you-go top-ups at $0.90 per 1,000 successful requests. For a Bing rank tracker doing a few thousand queries a month, that math usually beats paying for proxies plus CAPTCHA credits plus the engineer-hours to maintain selectors.

Which method should you choose?

Pick based on volume and how stable you need the output. Here is my decision rule.

• Learning or a one-off of under 50 queries: try Playwright locally, accept that some runs hit a CAPTCHA or a thin market, and do not build infrastructure for it.
• Any recurring job or commercial product: use a scraper API. Bing’s JS shell, session checks, IP limits, and CAPTCHA make DIY a maintenance treadmill, and a per-request API removes the proxy and CAPTCHA line items.
• You want grounded answers rather than rankings: Microsoft’s Grounding with Bing Search (inside Azure AI Agents) returns grounded responses for an AI agent. It returns no raw SERP, so it does not fit rank tracking or result-level scraping.

The deciding factor is that a plain HTTP request returns 0 results, as my test showed, and closing that gap yourself means owning a browser farm, a proxy pool, and a CAPTCHA solver. For most readers that is work an API has already done.