How to Scrape Twitter/X: A Python Guide

X (formerly Twitter) is still the largest stream of public real-time commentary on the web: breaking news, brand mentions, public-figure posts, sentiment. It is also a target that changed character in 2023, when the free API closed and the login wall tightened. I spent June 2026 testing what a plain Python scraper actually pulls from logged-out X. The result surprised me: it returns HTTP 200 and real tweet text, just not the engagement numbers most people want. This guide shows exactly what you can and cannot get, with the real output I captured, plus the scraper-API path for the fields the raw HTML withholds.

Can you legally scrape Twitter/X?

Scraping public X data is not automatically illegal in the US, and a recent court ruling favored scrapers, yet two other layers of risk remain. Treat all three separately.

The first layer is computer-crime law. US courts have generally held that accessing data open to the public without a login does not violate the Computer Fraud and Abuse Act, the reasoning the Ninth Circuit used in hiQ Labs v. LinkedIn in 2022. That logic applies to X’s public surfaces.

The second layer is X’s own contract, and here the news favors scrapers. In X Corp v. Bright Data Ltd. (N.D. Cal., case 3:23-cv-03698, filed 2023), X sued a data vendor over scraping. On May 9, 2024, Judge William Alsup dismissed X’s breach-of-contract, tortious-interference, and unjust-enrichment claims, holding that X had not shown a contract violation and warning that letting a platform dictate how public data is used could create “information monopolies that would disserve the public interest.” The dismissal came with leave to amend, so it is one strong data point, not a permanent shield. X’s Terms of Service still prohibit using “any robot, spider, scraper or other automated means to access the Services” without prior written consent, and X enforces account-level bans for logged-in automation regardless of the civil ruling. Note X published updated Terms taking effect January 15, 2026, so re-check the current version.

The third layer is data protection, and for X it is the one that bites. A tweet is personal data: it has an author, often names, opinions, location. The moment you store information about an EU resident, GDPR applies, and California’s CCPA/CPRA reaches residents there. I cover the general framing in my is web scraping legal guide. For X specifically, the defensible posture is narrow: collect public posts for aggregate analysis, avoid building profiles of named individuals, never log in a bot account, and get counsel before you store anything tied to identifiable people.

What data is available on a public X profile?

A logged-out X profile exposes a defined set of fields, but the raw HTML and the JavaScript-rendered view give you different subsets. The table below splits them, because that gap is the whole story of scraping X in 2026. “Static HTML” means what a plain requests call returns; “JS-rendered” means what loads after the browser runs X’s client code.

The practical line: text content and authorship are scrapable from the HTML, the numbers are not. If your use case is “what is this account saying,” plain requests gets you there. If it is “how much engagement did each post get,” you need rendering or an API.

Can you scrape X with Python requests? (Method 1: DIY)

Yes, partially. I tested it on June 12, 2026, and plain requests to public X profiles returns HTTP 200 with server-rendered tweet text, though not the engagement counts. This is the result you need to see before deciding whether the DIY path covers your use case. Here is the exact code I ran:

import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}

def scrape_profile(handle):
    url = f"https://x.com/{handle}"
    r = requests.get(url, headers=HEADERS, timeout=20)
    r.raise_for_status()
    soup = BeautifulSoup(r.text, "html.parser")

    og = lambda p: (soup.find("meta", property=p) or {}).get("content")
    profile = {
        "name": og("og:title"),
        "bio": og("og:description"),
        "avatar": og("og:image"),
    }
    # Tweets are server-rendered inside <article> blocks
    tweets = [a.get_text(" ", strip=True)
              for a in soup.find_all("article")
              if a.get_text(strip=True)]
    return profile, tweets

if __name__ == "__main__":
    prof, tweets = scrape_profile("nasa")
    print("name :", prof["name"])
    print("bio  :", prof["bio"])
    print("tweet blocks parsed:", len(tweets))
    for i, t in enumerate(tweets[:3], 1):
        print(f"[{i}] {t[:90]}")

And the real output it produced against x.com/nasa:

HTTP 200 OK
name : NASA (@NASA) on X
bio  : Making the seemingly impossible, possible. ✨
tweet blocks parsed: 13
--------------------------------------------------
[1] Pinned NASA @NASA Jun 9 Introducing Artemis III.

Four astronauts. Three launches. Two doc
[2] NASA @NASA Jun 11 We will hold a media teleconference at 11am ET on June 17 to preview the
[3] NASA's Johnson Space Center @NASAJohnson Jun 10 What do soccer and space have in common? ⚽

That is genuine data: 13 tweet blocks per page, each with author, handle, relative date, and full text. I confirmed the same shape (HTTP 200, 13 <article> blocks) on x.com/bbcworld and x.com/github, so it is reproducible, not a fluke for one account.

Now the limits, stated plainly so you do not waste a day building selectors that return nothing:

• No engagement numbers. The static HTML has zero data-testid="tweet" nodes and no <time datetime> attributes. Likes, reposts, replies, views, and exact timestamps are rendered by X’s client-side GraphQL after page load, so they are absent from what requests receives.
• No follower counts. Same reason: rendered client-side.
• Volume gets you blocked. This worked for a handful of requests from a residential IP. X’s robots.txt sets Disallow: / for the default user-agent and Crawl-delay: 1, and it explicitly disallows /*/followers, /*/following, /*/analytics, and the /status/.../likes and retweet paths. Datacenter IP ranges get throttled fast, and the HTML layout is undocumented and shifts, so selectors break without warning.

For exact timestamps and engagement, the DIY upgrade is a headless browser (Playwright) that runs X’s JavaScript. That works but is heavy: it needs guest tokens that expire every few hours, fresh residential IPs, and constant maintenance as X rotates its GraphQL operation IDs. At that point most teams hand the rendering problem to an API.

How does X block scrapers?

X blocks at several layers, and the defenses tightened after the 2023 API changes. Knowing each one tells you why plain requests gets text but not numbers, and why volume fails.

The 6,000 / 600 / 300 figures are the per-day read limits X applied in July 2023 for verified, unverified, and new unverified accounts. They constrain logged-in collection. A logged-out scraper runs into a different wall: the Crawl-delay of one second and IP-based throttling on datacenter ranges.

Scraping X with a scraper API (Method 2)

A scraper API removes two of the three walls at once: it renders the JavaScript so you get engagement counts and exact timestamps, and it routes the request through a rotating residential IP so you survive volume. You hand it the profile URL, it returns the fully rendered page, and the guest-token and IP-reputation problems become the vendor’s job.

ChocoData is one such service, with a universal endpoint plus a set of dedicated endpoints. Its request shape is a single GET carrying your target URL, your API key, and a flag to render JavaScript. Here is the pattern for sending an X profile URL through the universal endpoint:

import requests

API_KEY = "your_chocodata_key"

resp = requests.get(
    "https://api.chocodata.com/api/v1/universal",
    params={
        "api_key": API_KEY,
        "url": "https://x.com/nasa",
        "render_js": "true",   # run X's client code so counts/timestamps appear
        "country": "us",       # route via a US residential IP
    },
    timeout=90,
)

html = resp.text
# Now the rendered HTML contains the data-testid tweet nodes,
# time[datetime] timestamps, and engagement counts the static
# page omitted. Parse with BeautifulSoup as usual.

I did not run this snippet, because it needs a paid key, so treat the field names as illustrative and confirm the exact parameters against ChocoData’s own docs before you rely on them. The principle holds across any scraper API: you pass the URL, it returns the page from a rotating residential IP with JavaScript executed, and the rendering and detection problems stop being yours. ChocoData advertises residential proxy rotation, a universal endpoint, and 453 dedicated endpoints, and prices pay-as-you-go top-ups at $0.90 per 1,000 successful requests with JavaScript rendering billed as an add-on, which are the features and costs that matter against X’s rendering gate and IP fingerprinting.

DIY or scraper API: which should you use for X?

Pick by which fields you need and at what volume. The split is clean for X.

My honest read after testing: if you only need what accounts are posting, the DIY scraper above is real and free, and I would start there. The moment you need engagement metrics, full threads, or volume, the rendering and IP problems make a scraper API the lower-maintenance path. Whichever you choose, the legal limits do not move: public posts are still personal data, X’s Terms still ban automated access, and aggregate analysis is a far safer footing than profiling named individuals.

If you are new to the tooling, my BeautifulSoup guide covers the parsing side, the Python web scraping guide covers requests and sessions, and scraping without getting blocked covers proxies and headers. For the bigger picture, start at the web scraping pillar.