How to Scrape Google Scholar with Python

I tried to scrape Google Scholar the simple way and pasted the real output below. Short version: a single requests query returned HTTP 200 with 10 fully parseable results, but a 5-request burst from one IP tripped Google’s 429 CAPTCHA on the very first call. So DIY works for a few queries and falls over the moment you scale. This guide covers whether you can scrape Scholar, what each result holds, how the blocking fires, a working Python parser with real output, and the scraper-API path. For fundamentals, see my web scraping guide and the Python walkthrough.

Can you legally scrape Google Scholar?

Scraping public Scholar pages is a legal gray area, and you should read the source documents before writing code. Scholar’s robots.txt is the clearest technical signal. I fetched https://scholar.google.com/robots.txt on June 12, 2026 and it disallows the two paths you would actually scrape:

User-agent: *
Disallow: /search
Disallow: /index.html
Disallow: /scholar
Disallow: /citations?
Allow: /citations?user=

robots.txt is a crawling convention, not a statute, so it does not by itself create legal liability. The weight comes from elsewhere:

The hiQ ruling leaned toward scrapers for openly accessible data and held that scraping public pages is generally not a CFAA violation. It did not resolve breach-of-contract claims, and it is US-specific. A separate point matters for Scholar: the bibliographic facts themselves (title, authors, year, citation count) are facts, and facts are not copyrightable under Feist v. Rural Telephone (US Supreme Court, 1991). My rule of thumb: scrape only public result pages, never log in, rate-limit hard, store facts rather than copying abstracts wholesale, and keep author personal data out of what you retain. See scraping without getting blocked for the technical side. None of this is legal advice; if the data feeds a commercial product, talk to a lawyer.

What data can you extract from a Google Scholar result?

Each Scholar result is a single block (div.gs_r.gs_or.gs_scl) carrying bibliographic fields plus citation signals. Knowing the layout tells you what to target and what an API should return as JSON.

The div.gs_a string is the messy part. Authors, venue, and year arrive as one line that you split on the en-dash separators, and the format varies by source.

How does Google Scholar block scrapers?

Scholar fronts a rate limiter that watches IP, request cadence, and headers, then redirects suspect traffic to a CAPTCHA. I confirmed this in my own test. When I fired 5 rapid queries from a single IP with no delay, the first request returned HTTP 429 and its final URL landed on Google’s /sorry/ CAPTCHA path. Here is the raw signal from that burst:

req 0: status=429 len=3292 blocked_signal=True final_url_has_sorry=True
req 1: status=200 len=167295 blocked_signal=False
req 2: status=200 len=166618 blocked_signal=False
req 3: status=200 len=168024 blocked_signal=False
req 4: status=200 len=166825 blocked_signal=False

The defenses, in the order you hit them:

The behavior is probabilistic, not a clean threshold. One slow query usually succeeds; a tight burst tripped the 429 immediately. That is the core reason DIY scraping does not scale here without proxies.

Method 1: DIY scraping with Python (the tested result)

A plain requests + BeautifulSoup parser works for a single Scholar query, and I have the real output to prove it. The selectors are stable: each result is a div.gs_r.gs_or.gs_scl, the title sits in h3.gs_rt, and the citation count is the div.gs_fl link whose text starts with “Cited by”.

import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}

def scrape_scholar(query, start=0):
    url = "https://scholar.google.com/scholar"
    params = {"q": query, "hl": "en", "start": start}
    r = requests.get(url, params=params, headers=HEADERS, timeout=25)

    # Detect the rate-limit / CAPTCHA redirect before parsing.
    if r.status_code == 429 or "/sorry/" in r.url:
        raise RuntimeError(f"Blocked: HTTP {r.status_code} -> {r.url}")
    r.raise_for_status()

    soup = BeautifulSoup(r.text, "html.parser")
    results = []
    for block in soup.select("div.gs_r.gs_or.gs_scl"):
        title_el = block.select_one("h3.gs_rt")
        meta_el = block.select_one("div.gs_a")
        cited = None
        for a in block.select("div.gs_fl a"):
            if a.get_text(strip=True).startswith("Cited by"):
                cited = a.get_text(strip=True)
        results.append({
            "title": " ".join(title_el.get_text().split()) if title_el else None,
            "link": title_el.a["href"] if (title_el and title_el.a) else None,
            "meta": " ".join(meta_el.get_text().split()) if meta_el else None,
            "cited_by": cited,
        })
    return results

if __name__ == "__main__":
    for row in scrape_scholar("large language models")[:3]:
        print(row["title"][:60])
        print("  ", row["meta"][:70])
        print("  ", row["cited_by"])

I ran this against Scholar on June 12, 2026. The single query returned HTTP 200 and parsed cleanly. Real output, trimmed to three rows:

HTTP 200
A survey on evaluation of large language models
   Y Chang, X Wang, J Wang, Y Wu, L Yang... - ACM transactions on...,
   Cited by 6125
A survey of large language models
   WX Zhao, K Zhou, J Li, T Tang, Z Dong, Y Hou... - Frontiers of Comp...,
   Cited by 8848
ChatGPT for good? On opportunities and challenges of large language mo
   E Kasneci, K Sessler, S Kuechemann, M Bannert... - Learning and ind...,
   Cited by 10620

That is genuine Scholar data: titles, author/venue lines, and live citation counts. The catch is what you saw in the blocking section. The same code, fired in a tight loop, tripped a 429 on the first request. To run more than a handful of queries you need to add, at minimum:

• A delay between requests (several seconds, randomized). Scholar’s robots.txt sets no crawl-delay, yet the rate limiter clearly rewards slower cadence.
• Rotating residential proxies, because a single IP gets a CAPTCHA fast.
• A CAPTCHA fallback, since once /sorry/ fires the IP is parked until the challenge is solved.
• Header rotation and retry-with-backoff on every 429.

At that point you are building a small scraping infrastructure project, which is the trade-off the next method removes.

Method 2: Scraping Google Scholar with a scraper API

A scraper API turns the whole problem into one HTTP call that returns the page, which is why teams skip the proxy-and-CAPTCHA stack. The API runs the request from a clean residential IP, rotates on failure, clears the /sorry/ challenge, and hands back the HTML you parse with the same BeautifulSoup selectors from Method 1.

ChocoData (chocodata.com) is one example I checked against its own docs. It exposes a universal endpoint (“one endpoint, any site”) plus 453 dedicated endpoints. For Scholar, the universal endpoint is the route, since there is no dedicated Scholar endpoint to assume. The documented request shape:

import requests
from bs4 import BeautifulSoup

API_KEY = "YOUR_API_KEY"  # free tier: 1,000 requests/month, no card

resp = requests.get(
    "https://chocodata.com/api/v1/universal",
    params={
        "url": "https://scholar.google.com/scholar?q=large+language+models&hl=en",
        "render": "true",     # run the headless browser
        "api_key": API_KEY,
    },
    timeout=60,
)
resp.raise_for_status()

# The API handles the IP rotation and CAPTCHA; you parse the returned HTML
# with the exact selectors from Method 1.
soup = BeautifulSoup(resp.text, "html.parser")
results = soup.select("div.gs_r.gs_or.gs_scl")
print(len(results), "results")

The endpoint pattern is GET /api/v1/universal?url=... for any page, and GET /api/v1/{site}/{resource}?api_key=... for dedicated endpoints. Per ChocoData’s site, the service handles residential proxy rotation, CAPTCHAs, failed-request retries, and optional JavaScript rendering. I have not independently benchmarked their Scholar success rate, so treat the parsing and success-rate claims as vendor-stated until you test on the free tier. The honest pitch is narrow: you are paying the API to absorb the 429-and-CAPTCHA problem that killed the DIY loop, not for magic parsing.

Which method should you choose?

Match the method to your query volume, since that is the variable Scholar punishes. Below a dozen lookups, DIY is free and works, as my test showed. Past that, the blocking cost dominates.

My take: write the Method 1 parser first because it teaches you the DOM and proves the selectors, then reach for an API the moment you need consistent volume. The selectors are identical across both, so the migration is a one-line change to where you fetch the HTML. If you are building the parser from scratch, my BeautifulSoup guide covers the selector patterns, and the Scrapy guide covers running this at scale with built-in throttling and retries.