How to Scrape Yandex Search Results

I tried to scrape Yandex the simple way, and Yandex stopped me at the door. A plain Python requests call to the search page did not even return a results page: it returned a 302 redirect to /showcaptchafast, Yandex’s CAPTCHA wall, with the response header X-Yandex-Captcha: captcha. I am pasting that real output below. This guide covers whether you can scrape Yandex, what the results page holds, the exact block you will hit, the official XML API, and the scraper-API path that returns usable data. For the fundamentals, see my web scraping guide and the Python scraping walkthrough.

Can you legally scrape Yandex search results?

Scraping public SERPs is a contested gray area, and for Yandex you should read three documents before writing any code. Yandex’s robots.txt is the clearest signal. I fetched it on June 12, 2026 and it disallows the search paths directly:

User-agent: *
...
Disallow: /search
Disallow: /xmlsearch
Disallow: /yandsearch
Disallow: /sitesearch
Disallow: /video/search

robots.txt is a crawling convention, not a statute, so it does not by itself create legal liability. The legal weight comes from elsewhere:

Yandex offers a sanctioned alternative most search engines do not advertise as loudly: an official Search API that returns results in XML. That matters legally, because there is a documented, paid way to get Yandex results without touching the HTML path that robots.txt blocks. My rule of thumb for the HTML SERP: scrape only public result pages, never log in, rate-limit hard, and keep personal data out of what you store. For deeper coverage of blocking and ethics, see scraping without getting blocked. None of this is legal advice; if the data feeds a commercial product, talk to a lawyer.

What data can you extract from a Yandex SERP?

A Yandex results page is a stack of distinct blocks, and each block carries different fields. Knowing the layout tells you what to target and what an API should return as structured data.

Two of these decide your method. The image/video packs and some entity panels are populated by JavaScript, so a static HTML fetch would miss them even if Yandex served you the base markup. Combined with the CAPTCHA wall, that pushes any serious attempt toward either the official API or a real browser behind proxies.

How does Yandex block scrapers?

Yandex blocks at the front door with SmartCaptcha, its own anti-bot product, then escalates on IP and behavior. Here is what I observed and what comes after it.

When I requested the search page with requests and a desktop Chrome user agent, Yandex did not return a results page at all. It returned an HTTP 302 redirect whose Location pointed at /showcaptchafast, and the response carried the header X-Yandex-Captcha: captcha. Following the redirect (curl -L) landed on a 39 KB CAPTCHA page, not a SERP. I tested both yandex.com and yandex.ru and got the same redirect on each.

The defenses stack like this:

1. SmartCaptcha redirect - the search path 302-redirects datacenter and bot-like clients to /showcaptchafast before any results render.
2. Browser fingerprint checks - Yandex requests a long list of client hints (Sec-CH-UA-*, viewport, device memory) via Accept-CH; a bare HTTP client supplies none of them.
3. Cookie and session checks - the first response sets yandexuid, i, yashr, and other cookies; repeat requests without sane session behavior look automated.
4. IP reputation and rate-limiting - bursts from a single datacenter IP get challenged or throttled fast.
5. Region variance - results change by the lr (region) and lang parameters and by the IP’s location, so output is unstable without geo control.

So a scraper has to clear the CAPTCHA, present a believable browser fingerprint, and come from a residential IP. That is three problems before you parse a single result.

Method 1: DIY scraping with Python (the honest result)

A plain requests + BeautifulSoup script never reaches a results page, and here is the code plus the real output. I ran this on June 12, 2026 with Python 3.13.7 and requests 2.34.2.

import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}

def scrape_yandex(query, page=0):
    url = "https://yandex.com/search/"
    params = {"text": query, "p": page}
    r = requests.get(url, params=params, headers=HEADERS,
                     timeout=20, allow_redirects=False)
    print("HTTP", r.status_code)
    if r.status_code == 302 and "showcaptcha" in r.headers.get("Location", ""):
        print("Blocked: Yandex SmartCaptcha challenge")
        return []
    soup = BeautifulSoup(r.text, "html.parser")
    results = []
    for item in soup.select("li.serp-item"):
        link = item.select_one("a.organic__url, a.Link")
        title = item.select_one("h2 .organic__title-wrapper, h2")
        if link and title:
            results.append({"title": title.get_text(strip=True),
                            "url": link.get("href")})
    return results

if __name__ == "__main__":
    rows = scrape_yandex("web scraping python")
    for i, row in enumerate(rows, 1):
        print(i, row["title"], row["url"])
    print("Parsed", len(rows), "results")

The real output:

HTTP 302
Blocked: Yandex SmartCaptcha challenge
Parsed 0 results

The 302 is the whole story: Yandex redirects to its CAPTCHA before serving any results, so there is nothing to parse. The li.serp-item selector below the guard never runs against a real SERP. To actually reach a results page you would need a real browser plus a residential IP. A headless browser like Playwright can render the page and even display the CAPTCHA, but it does not solve the CAPTCHA for you. Install with pip install playwright then playwright install chromium.

from playwright.sync_api import sync_playwright

def scrape_yandex_browser(query: str):
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page(
            user_agent=(
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                "AppleWebKit/537.36 (KHTML, like Gecko) "
                "Chrome/126.0.0.0 Safari/537.36"
            ),
            locale="en-US",
        )
        page.goto(
            f"https://yandex.com/search/?text={query}",
            wait_until="domcontentloaded",
        )
        # If this selector times out, you have hit /showcaptchafast.
        page.wait_for_selector("li.serp-item", timeout=8000)

        results = []
        for block in page.query_selector_all("li.serp-item"):
            title_el = block.query_selector("h2")
            link_el = block.query_selector("a.Link, a.organic__url")
            if title_el and link_el:
                results.append({
                    "title": title_el.inner_text(),
                    "url": link_el.get_attribute("href"),
                })
        browser.close()
        return results

if __name__ == "__main__":
    for i, row in enumerate(scrape_yandex_browser("web+scraping+python"), 1):
        print(i, row["title"], "->", row["url"])

I am not pasting output for the Playwright version, because from a plain datacenter IP it lands on the same /showcaptchafast wall and wait_for_selector times out. That is the honest state of DIY Yandex scraping. To make it production-grade you would add: residential proxy rotation, a SmartCaptcha solver, realistic browser-hint headers, per-region lr/lang handling, and selector maintenance whenever Yandex reshuffles its markup. For browser-automation fundamentals see the Scrapy guide and BeautifulSoup guide.

DIY costs at scale add up fast:

Method 2: The official Yandex Search API (XML)

Yandex sells a licensed way to get search results, and it sidesteps the CAPTCHA entirely. The Yandex Search API returns results in XML for registered users; you create a Yandex Cloud account, get a folder ID and an API key, and query an endpoint that hands back ranked results in a documented schema. There is also a newer mode that returns JSON and generative answers.

The trade-offs, from Yandex’s own documentation:

Use this when you want results Yandex itself authorizes and you can live with its XML schema and daily cap. It will not give you the exact pixel-for-pixel HTML SERP (ad placements, Turbo badges, some packs), and the regional defaults are controlled by API parameters rather than a browser’s IP. For raw SERP fidelity or higher volume, a scraper API is the other route.

Method 3: Scraping Yandex with a scraper API

A scraper API turns the whole problem into one HTTP call that returns data, which is why most teams skip both the browser stack and the XML schema. The API runs the headless browser, rotates residential IPs, clears the SmartCaptcha, and hands back the rendered page or parsed fields. You send a URL, you get usable output.

ChocoData (chocodata.com) is one example I checked against its own docs. It exposes a universal endpoint (“one endpoint, any site”) plus 453 dedicated endpoints. For Yandex, the universal endpoint with JavaScript rendering is the straightforward route. The documented request shape:

import requests

API_KEY = "YOUR_API_KEY"  # free tier: 1,000 requests/month, no card

resp = requests.get(
    "https://chocodata.com/api/v1/universal",
    params={
        "url": "https://yandex.com/search/?text=coffee+maker",
        "render": "true",          # run the headless browser
        "country": "tr",           # residential exit, e.g. Turkey
        "api_key": API_KEY,
    },
    timeout=60,
)
data = resp.json()  # rendered HTML or parsed fields, depending on options

The endpoint pattern is GET /api/v1/{site}/{resource}?api_key=... for dedicated endpoints, and GET /api/v1/universal?url=... for any page. Per ChocoData’s site, the service handles residential proxy rotation, CAPTCHAs, failed-request retries, and optional JavaScript rendering. I have not independently benchmarked their Yandex success rate, so treat their parsing claims as vendor-stated until you test on the free tier.

How the three methods compare on the things that actually matter:

ChocoData’s published pricing: a free tier of 1,000 requests/month with no card, then paid plans from $19/month (27,000 requests) up, and pay-as-you-go top-ups at $0.90 per 1,000 successful requests. For a Yandex rank tracker doing a few thousand queries a month, that math usually beats paying for proxies plus CAPTCHA credits plus the engineer-hours to maintain selectors.

Which method should you choose?

Pick based on what you need from the data and how much volume you run. Here is my decision rule.

• You want results Yandex authorizes and can accept an XML schema: use the official Yandex Search API. It is the compliant feed, with a folder ID, an API key, and a daily query cap.
• You need the live HTML SERP (ad slots, Turbo badges, packs) or higher volume: use a scraper API. The SmartCaptcha wall, browser-hint checks, and IP reputation make DIY a maintenance treadmill, and a per-request API removes the proxy and CAPTCHA line items.
• Learning or a one-off of a handful of queries: try Playwright locally, but expect the /showcaptchafast wall on a datacenter IP and do not build infrastructure for it.

The deciding factor is the one my test showed: a plain HTTP request to Yandex returns a 302 to a CAPTCHA, not a SERP. Closing that gap yourself means owning a browser farm, a proxy pool, and a CAPTCHA solver. For most readers that is work either the official API or a scraper API has already done.