How to Scrape Real Estate Sites: A Rightmove Scraper in Python

Real estate sites are some of the most structured data on the public web: every listing has a price, bed count, address, and agent, repeated thousands of times. That regularity makes them worth scraping and makes the sites defensive about it. This guide builds a working Rightmove scraper in Python, shows the real output I got, and covers what breaks when you scale. The same approach maps to Zoopla, Idealista, Crexi, and most portals.

If you are new to the tooling, start with my Python web scraping guide and the BeautifulSoup tutorial, then come back here.

Is it legal to scrape Rightmove?

Scraping public listing pages is generally lawful, with conditions you need to respect. Three facts shape the answer for Rightmove specifically.

First, the data is public: anyone can load a search page without logging in. Courts in the US (hiQ v. LinkedIn) and the position taken by regulators in the EU treat scraping of publicly available data as distinct from unauthorised access. Second, Rightmove’s own terms reserve the content and prohibit systematic copying for a competing service, so what you do with the data matters more than the act of fetching it. Third, individual listings can contain personal data (agent names, sometimes vendor detail), which brings UK GDPR into scope if you store it.

Practical rules I follow:

• Scrape only public pages. Never log in and scrape behind an account.
• Respect robots.txt (checked below) and rate-limit hard.
• Use the data for analysis, pricing models, or lead research, not to clone the listings into a rival portal.
• Drop or anonymise personal fields you do not need.

This is not legal advice. For the wider picture, see is web scraping legal. When in doubt about a commercial use, ask a solicitor.

What does robots.txt allow on Rightmove?

The search results path and property detail pages are allowed; API and map endpoints are blocked. I fetched the live file:

curl -s https://www.rightmove.co.uk/robots.txt

User-agent: *
Disallow: /*/fullscreen/image-gallery.html*
Disallow: /*/mortgage-calculator/*
Disallow: /api/*
Disallow: /property-for-sale/draw-a-search.html?*
Disallow: /property-for-sale/map.html?*
Disallow: /student-accommodation/find.html*
...
Sitemap: https://www.rightmove.co.uk/sitemap.xml

What this means for a scraper:

• /property-for-sale/find.html (the search results page) is not disallowed for User-agent: *.
• /properties/<id> detail pages are not disallowed.
• /api/*, the draw-a-search and map views, and fullscreen galleries are off limits.
• Named crawlers get their own blocks. GPTBot, MJ12bot, TrovitBot, and others are called out by name, so identifying yourself as one of those gets you denied by policy.

robots.txt is a rule you choose to honour, not a wall. Honouring it keeps you on the right side of the site’s stated wishes and out of the paths most likely to trip server-side defences.

What data can you get from a Rightmove listing?

Each search result carries the core listing fields plus geo coordinates and agent detail. Here is what the embedded JSON exposes per property, with the field names from the live page:

Detail pages (/properties/<id>) add the full description, floorplans, EPC, station distances, and the price history, stored in a separate window.PAGE_MODEL blob.

How does Rightmove block scrapers?

Rightmove serves clean HTML to a first request, then throttles by IP and behaviour as you scale. From my testing and the page structure, the defences stack up like this:

The single biggest gotcha: the listing data is not in the HTML you would inspect by eye. It sits in a __NEXT_DATA__ JSON object that Next.js uses to hydrate the page. That is good news. JSON is far more stable to parse than CSS class names that change on every redeploy. For the general playbook, see scraping without getting blocked.

Method 1: A DIY Rightmove scraper in Python

You can scrape a Rightmove search page with requests alone, because the data ships in the page as JSON. No Selenium, no Playwright. Here is the scraper I ran.

import json
import re
import requests

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-GB,en;q=0.9",
}

# locationIdentifier REGION^87490 = London.
# Find yours by searching on the site and reading the URL.
URL = (
    "https://www.rightmove.co.uk/property-for-sale/find.html"
    "?searchType=SALE&locationIdentifier=REGION%5E87490"
)


def scrape_rightmove(url):
    resp = requests.get(url, headers=HEADERS, timeout=30)
    resp.raise_for_status()

    # Rightmove is a Next.js app: listing data sits in a __NEXT_DATA__ JSON blob.
    match = re.search(
        r'id="__NEXT_DATA__" type="application/json">(.*?)</script>',
        resp.text,
        re.S,
    )
    data = json.loads(match.group(1))
    results = data["props"]["pageProps"]["searchResults"]

    rows = []
    for p in results["properties"]:
        rows.append({
            "id": p["id"],
            "price": p["price"]["displayPrices"][0]["displayPrice"],
            "bedrooms": p["bedrooms"],
            "type": p["propertySubType"],
            "address": p["displayAddress"],
            "agent": p["customer"]["brandTradingName"],
            "added": p["listingUpdate"].get("listingUpdateDate", ""),
        })
    return results["resultCount"], rows


if __name__ == "__main__":
    total, rows = scrape_rightmove(URL)
    print(f"total matches: {total}")
    print(f"parsed on this page: {len(rows)}\n")
    for r in rows[:8]:
        print(f"{r['id']:>15} | {r['price']:>12} | {r['bedrooms']}bed | "
              f"{r['type']:<14} | {r['address'][:40]}")

I ran this on 12 June 2026 against the live London search page. Real output:

total matches: 62,167
parsed on this page: 25

      170001023 |     £995,000 | 4bed | Semi-Detached  | Warwick Road, Bounds Green, London, N11
      148711631 |  £60,000,000 | 5bed | Apartment      | One Hyde Park, Knightsbridge, London, SW
757097812420945 |          POA | 0bed | Land           | Vincent House, 5 Pembridge Square, Londo
      155320229 |  £49,950,000 | 10bed | House          | Avenue Road, St Johns Wood, London, NW8
      171185315 |  £49,500,000 | 7bed | End of Terrace | Balfour Place, Mayfair, London, W1K
      152205953 |  £47,000,000 | 6bed | House          | Whistler Square, Chelsea Barracks, Londo
      152236991 |  £47,000,000 | 7bed | Town House     | Whistler Square, London, SW1W
      158528285 |  £46,000,000 | 5bed | Apartment      | One Hyde Park, Knightsbridge, London, SW

That is the genuine response: a 200, 62,167 total matches, 25 listings parsed from the first page. Notes from the run:

• The first request worked with one header. That does not last. Loop over pages fast and you start collecting 429s.
• Rightmove paginates in steps of 24 via an &index= parameter (index=24, index=48, …). The site historically caps results around index 1008, so you cannot walk all 62,167 from one search; you split by region or price band.
• The price field can be "POA" (price on application), so cast carefully before doing maths on it.
• For detail pages, swap the regex for the window.PAGE_MODEL assignment. That blob is larger and assigned in script, so extract it with a more careful pattern (capture from the first { to the matching close) rather than the __NEXT_DATA__ selector above.

This is the cheap, honest version: it works for a few hundred listings from one machine. The DIY ceiling is the IP. For the framework-based version of this skeleton, see Scrapy.

Method 2: Scrape Rightmove with a scraper API

When the DIY route starts hitting 429s, a scraper API takes over proxy rotation, retries, and geo-routing so your code stays a parser. You keep the exact same JSON-extraction logic; you only change how you fetch. I use ChocoData for this. It has a universal endpoint plus 453 dedicated endpoints, and the universal one is what you want for Rightmove since there is no dedicated real estate route.

The universal endpoint, from their docs:

GET https://api.chocodata.com/api/v1/universal/get
    ?api_key=cd_live_YOUR_KEY
    &url=<target url>
    &country=gb

Drop it into the same scraper. The only change is the request line:

import json
import re
import requests

API_KEY = "cd_live_YOUR_KEY"
TARGET = (
    "https://www.rightmove.co.uk/property-for-sale/find.html"
    "?searchType=SALE&locationIdentifier=REGION%5E87490"
)


def scrape_via_chocodata(target_url):
    resp = requests.get(
        "https://api.chocodata.com/api/v1/universal/get",
        params={
            "api_key": API_KEY,
            "url": target_url,
            "country": "gb",  # force a UK egress IP
        },
        timeout=90,
    )
    resp.raise_for_status()

    # Same parse as the DIY version: the JSON blob is still in the HTML.
    html = resp.text
    match = re.search(
        r'id="__NEXT_DATA__" type="application/json">(.*?)</script>',
        html,
        re.S,
    )
    data = json.loads(match.group(1))
    return data["props"]["pageProps"]["searchResults"]["properties"]


if __name__ == "__main__":
    for p in scrape_via_chocodata(TARGET)[:5]:
        price = p["price"]["displayPrices"][0]["displayPrice"]
        print(p["id"], price, p["displayAddress"][:45])

What the API buys you against Rightmove’s defences:

Two honest caveats. The country param routing and proxy rotation are the load-bearing features here; on ChocoData the render_js and screenshot parameters are documented as reserved and return 501 not_implemented today. For Rightmove that is fine, because the data is already in the static HTML as JSON, so you do not need a rendered browser. If you were scraping a portal that builds listings client-side with no embedded JSON, you would want a provider whose JS rendering is live, and you would reach for Playwright on the DIY side.

Which method should you use?

Use plain requests for small jobs and a scraper API once IP blocks become your bottleneck. The decision comes down to volume and tolerance for maintenance.

My rule: prototype with Method 1 to confirm the parse works (it does, the output above is real), then switch the fetch to Method 2 the moment you need more than a few pages or want it to run unattended. The parser stays identical, which is the whole point of keeping the JSON-extraction logic separate from the fetch.

Honest limits

The search page is easy; full-scale, every-listing extraction is not. Set expectations before you build a pipeline on this:

• One search returns at most ~1008 results regardless of total matches. You must segment by region, price, or property type to cover a market.
• Detail pages cost a request each. Pulling descriptions and price history for 10,000 listings is 10,000 fetches plus the search fetches.
• The embedded JSON schema is owned by Rightmove. They can rename searchResults or restructure __NEXT_DATA__ in any deploy, and your regex path breaks. Pin nothing you cannot re-find in five minutes.
• Personal data (agent names, occasionally vendor info) is in scope for UK GDPR if you store it. Keep only the fields you actually use.

The mechanics here transfer to most portals: find where the site hides its JSON, fetch the page, parse the blob, and move the fetch behind a scraper API when the IP becomes the limit.