How to Scrape Reddit: A Python Guide

Reddit is one of the richest public text datasets on the web: 100,000+ active communities, structured threads, votes, and timestamps. It is also one of the harder targets to scrape with a naive script in 2026. The old “append .json to any URL” trick is blocked for server-side requests, and the HTML now ships behind a JavaScript verification challenge. I tested both against live Reddit in June 2026 and pasted the real responses below. This guide shows what fails, then the two methods that actually work: the official OAuth Data API for moderate volume, and a scraper API for scale.

Can you scrape Reddit legally?

You can scrape public Reddit content, but the terms are stricter than they used to be and the technical barriers are real. Reddit’s Data API Terms allow access to public posts and comments through its API, and public pages are visible without login. The well-known caveats: the terms prohibit using Reddit data to train machine-learning models or build a competing product without a separate commercial agreement, and they prohibit scraping private or deleted content. The 2023 API policy change that introduced paid pricing was aimed squarely at high-volume commercial extraction.

Practically, the safe lane is: authenticate through the official API, pull only public content, stay inside the rate limit, and do not resell or train on the data. That keeps you on the right side of both the terms and Reddit’s anti-bot systems. For a wider primer on the legal and technical basics, see my web scraping guide.

What data can you get from Reddit?

The public API and pages expose the full structure of a subreddit and its threads. Here is what is available and where it lives:

Private subreddits, removed comments, vote-by-vote breakdowns, and individual viewer identities are not available. The API gives you aggregate scores, not who voted.

Why does a plain requests scraper fail on Reddit?

Because Reddit blocks unauthenticated server traffic at two layers: the data endpoint and the HTML. I ran the obvious approach against live Reddit in June 2026 with a real browser User-Agent. Here is the exact script:

import requests

UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
      "(KHTML, like Gecko) Chrome/124.0 Safari/537.36")

# Attempt 1: the public JSON endpoint
r = requests.get("https://www.reddit.com/r/python.json?limit=3",
                 headers={"User-Agent": UA}, timeout=20)
print(f"[JSON]  GET /r/python.json   -> {r.status_code} {r.reason}, {len(r.content)} bytes")

# Attempt 2: the rendered HTML listing
r2 = requests.get("https://www.reddit.com/r/python/",
                  headers={"User-Agent": UA}, timeout=20)
title = ""
if "<title>" in r2.text:
    title = r2.text.split("<title>")[1].split("</title>")[0].strip()
print(f"[HTML]  GET /r/python/        -> {r2.status_code} {r2.reason}, {len(r2.content)} bytes")
print(f"        <title>: {title!r}")
print(f"        post elements found: {r2.text.count('shreddit-post')}")

The real output:

[JSON]  GET /r/python.json   -> 403 Blocked, 189908 bytes
[HTML]  GET /r/python/        -> 200 OK, 8433 bytes
        <title>: 'Reddit - Please wait for verification'
        post elements found: 0

Two takeaways. The .json endpoint returns a hard 403 Blocked from a datacenter IP no matter what User-Agent I send. The HTML page returns 200 OK, which looks like success, but the body is 8.4 KB titled “Reddit - Please wait for verification”: a JavaScript challenge that runs in a browser before any content loads. There are zero shreddit-post elements, so BeautifulSoup has nothing to parse.

Here is how Reddit blocks each naive method:

The cleanest fix is to stop fighting the anti-bot layer and authenticate. That is what the official API is for.

Method 1: Scrape Reddit with the official OAuth API

Register an app, authenticate with OAuth2, and you get clean JSON at 60 requests per minute for free. This is the method Reddit endorses and the one that does not get blocked. Per Reddit’s official API rules, clients must authenticate with OAuth2, may make up to 60 requests per minute, and must send a unique, descriptive User-Agent in the format <platform>:<app ID>:<version string> (by /u/<username>).

Step 1: Register an app

Log in to Reddit, go to your app preferences, and create a new app of type “script”. Reddit gives you a client_id (under the app name) and a client_secret. These are the credentials your script authenticates with.

Step 2: Call the API

The community-standard library is PRAW (Python Reddit API Wrapper). It handles the OAuth handshake, pagination, and the X-Ratelimit-* headers for you. Install it with pip install praw, then a read-only client is a few lines:

import praw

reddit = praw.Reddit(
    client_id="YOUR_CLIENT_ID",
    client_secret="YOUR_CLIENT_SECRET",
    user_agent="windows:bestscraperapi.reddit-demo:v1.0 (by /u/your_username)",
)

for post in reddit.subreddit("python").hot(limit=5):
    print(f"{post.score:>6}  {post.title}")
    print(f"        by u/{post.author} | {post.num_comments} comments")

This authenticates, requests the hot listing of r/python, and prints score, title, author, and comment count for the top five posts. PRAW lazily paginates, so iterating past limit=5 keeps fetching pages until you stop, throttling to stay under 60 requests per minute.

I am not pasting output for this block because it requires my personal Reddit credentials, which I will not commit to a public article. The script above is the exact, working call. Swap in your own client_id, client_secret, and username and it returns live data.

If you prefer raw requests

You do not need PRAW. The OAuth flow with plain requests is two calls: get a token, then send it as a bearer header.

import requests

auth = requests.auth.HTTPBasicAuth("YOUR_CLIENT_ID", "YOUR_CLIENT_SECRET")
headers = {"User-Agent": "windows:bestscraperapi.reddit-demo:v1.0 (by /u/your_username)"}

# 1. Get an app-only OAuth token
token_res = requests.post(
    "https://www.reddit.com/api/v1/access_token",
    auth=auth,
    data={"grant_type": "client_credentials"},
    headers=headers,
    timeout=20,
)
token = token_res.json()["access_token"]

# 2. Call the API on the oauth.reddit.com host with the bearer token
headers["Authorization"] = f"bearer {token}"
res = requests.get(
    "https://oauth.reddit.com/r/python/hot",
    headers=headers,
    params={"limit": 5},
    timeout=20,
)
for child in res.json()["data"]["children"]:
    p = child["data"]
    print(p["score"], "-", p["title"])

Note the host is oauth.reddit.com, not www.reddit.com. The bearer token is what gets you past the 403 I hit earlier. This is the same path PRAW automates.

The limits of Method 1

The free tier is generous for research and small projects, and painful at scale. The constraints:

• 60 requests per minute. Each request returns up to 100 items, so roughly 6,000 items/minute, but a deep crawl of comment trees burns requests fast.
• Paid above the free tier. Reddit’s 2023 pricing set high-volume commercial access at $0.24 per 1,000 API calls. A million-call job is $240.
• Credentials are a chokepoint. Every request ties back to your app. Abuse the rate limit and Reddit can suspend the app.
• No historical archive. The API serves recent and current data, not a full back-catalog of a subreddit’s history.

For moderate volume and clean code, Method 1 is the right answer. When you need scale, want to skip OAuth entirely, or hit a JS-rendered page the API does not cover, a scraper API is the better tool.

Method 2: Scrape Reddit with a scraper API

A scraper API takes the proxy rotation, the verification challenge, and the IP blocking off your plate: you send a target URL, it returns the content. This is the path when you need volume or do not want to manage OAuth credentials and rate-limit accounting. ChocoData is one such service, with a universal endpoint plus 453 dedicated endpoints for specific sites.

The pattern is one request to the API with your target URL:

import requests

API_KEY = "YOUR_CHOCODATA_KEY"

res = requests.get(
    "https://api.chocodata.com/v1",
    params={
        "api_key": API_KEY,
        "url": "https://www.reddit.com/r/python/hot/",
        "render_js": "true",   # execute the verification challenge
    },
    timeout=70,
)
print(res.status_code)
print(res.text[:500])

The service rotates the request through a residential IP, runs the JavaScript challenge that returned an empty page for my plain requests call, and hands back the rendered HTML or JSON. You then parse it with BeautifulSoup as normal. I have not run this block here because it needs a paid API key, so I am not stamping this article as tested against ChocoData. The code reflects the standard scraper-API request shape; confirm the exact parameter names against the provider’s current docs.

Here is how the two working methods compare for Reddit specifically:

Which method should you use?

Pick based on volume and how much infrastructure you want to own. For a research project, a dashboard, or a hobby bot pulling a few thousand items, the official OAuth API is free, clean, and endorsed by Reddit: use Method 1. For high-volume extraction, monitoring many subreddits, or pages that only render in a browser, the OAuth rate limit and per-call pricing become the bottleneck, and a scraper API that absorbs the proxy and anti-bot work is the better fit: use Method 2.

The one thing to avoid is the naive approach I tested at the top. Plain requests against the .json endpoint or the HTML page returns a 403 or an empty verification page in 2026. Authenticate or delegate, do not scrape Reddit raw.

To go deeper on the building blocks, see my guides on scraping with Python, parsing HTML with BeautifulSoup, Scrapy for larger crawls, and scraping without getting blocked.