How to Scrape Walmart (Python Guide)

Walmart.com is one of the most-requested retail scraping targets: live prices, stock by store, ratings, and seller data across millions of SKUs. It is also gated by one of the more deceptive anti-bot setups, because it answers bots with a 200, not a 403. This guide builds a real Python scraper, runs it against a live Walmart page, and shows the exact response I got. I ran the code in June 2026 and pasted the real output below, including the CAPTCHA that stopped it. Then I cover the route that actually returns data.

Can you scrape Walmart, and is it legal?

You can send a request to a public Walmart page, but Walmart blocks bots with a CAPTCHA challenge and its terms restrict automated access. The legality and the technical reality are two separate questions, so I will split them. The technical reality (next section) is that a naive scraper gets a 200 that contains no data.

On US computer-access law, reading pages anyone can load without logging in sits on the safer side of the line. In hiQ Labs v. LinkedIn, the Ninth Circuit held that scraping public profile data does not violate the Computer Fraud and Abuse Act, and the Supreme Court’s Van Buren decision narrowed the CFAA toward genuine hacking. Fetching a public product page with a script is not, on its own, a federal computer crime.

Contract and property law are where Walmart pushes back. Walmart’s Terms of Use prohibit using any robot, spider, scraper, or other automated means to access the site and to monitor or copy its content without written permission. Its robots.txt (covered below) disallows the search path outright. Ignoring those is a breach of the site terms and, under the eBay v. Bidder’s Edge line of cases, aggressive automated querying against the operator’s wishes can support a trespass-to-chattels claim. Stay on public pages, never script a logged-in account, do not collect personal data (that pulls in privacy law), and keep request volume low. I am a tester, not a lawyer, so get advice before a commercial scrape. My web scraping pillar covers the legal picture in full.

What does Walmart’s robots.txt say?

Walmart’s robots.txt disallows the search path I tested and many API and store endpoints. I fetched it live in June 2026 (HTTP 200, 3,580 bytes). Here are the directives that matter for a product or search scraper:

User-agent: *
Disallow: /account/
Disallow: /api/
Disallow: /search
Disallow: /typeahead/
Disallow: /search/search-ng.do
Disallow: /store/category/
Disallow: /store/*/search
Disallow: /orders
Allow:    /reviews/product/
Allow:    /reviews/seller/
...
User-agent: Adsbot-Google
User-agent: Mediapartners-Google
User-agent: Slurp
Crawl-delay: 5

Two things stand out. Disallow: /search covers the exact /search?q= URL most scrapers hit first, so the search results path is off-limits by the file itself. The Crawl-delay: 5 only appears for a named group (Google’s ad bots and Yahoo’s Slurp), not for User-agent: *. Product pages under /ip/ are not in the disallow list, but as the next section shows, the server blocks them with a CAPTCHA regardless of what robots.txt permits. robots.txt is not law, and ignoring it is both evidence of intent in a dispute and a breach of the terms that incorporate it.

What data is on a Walmart product or search page?

A search results page and a product page together carry the commercially useful fields. Walmart renders with Next.js and embeds most of this data as JSON inside the page (a __NEXT_DATA__-style script block) in addition to the visible HTML. Here is what is available, assuming you can retrieve the real page (the next section shows why a plain scraper cannot):

The cleanest path on Walmart is the embedded JSON, not scraping individual DOM nodes, because the price and stock are server-rendered into that JSON blob. Retrieval is the problem, not parsing. A script never reaches the JSON because the response it gets is a challenge page.

Method 1: scrape Walmart with requests and BeautifulSoup (what really happened)

A requests GET with a real browser User-Agent plus BeautifulSoup is the standard first attempt, and on Walmart it returns a 200 that contains a CAPTCHA instead of data. Here is the exact code I ran against a live Walmart search page and a live product page:

import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/124.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Sec-Fetch-Mode": "navigate",
}

urls = {
    "search":  "https://www.walmart.com/search?q=coffee+maker",
    "product": ("https://www.walmart.com/ip/"
                "Keurig-K-Mini-Single-Serve-K-Cup-Pod-Coffee-Maker-Black/356659277"),
}

for label, url in urls.items():
    resp = requests.get(url, headers=HEADERS, timeout=25)
    soup = BeautifulSoup(resp.text, "html.parser")
    title = soup.find("title")
    print(f"[{label}] STATUS:", resp.status_code)
    print(f"[{label}] BYTES :", len(resp.content))
    print(f"[{label}] TITLE :", title.get_text(strip=True) if title else None)
    print(f"[{label}] PRICES:", resp.text.count("$"))

This is the real output I got, run against both URLs:

[search]  STATUS: 200
[search]  BYTES : 15195
[search]  TITLE : Robot or human?
[search]  PRICES: 0
[product] STATUS: 200
[product] BYTES : 15195
[product] TITLE : Robot or human?
[product] PRICES: 0

Both URLs returned HTTP 200, which is the trap. A scraper that checks only status_code == 200 would log success and move on. The body is a 15,195-byte page titled “Robot or human?” with zero $ characters, so there are no prices and no product fields to parse. I inspected the body and confirmed the signature of the block: it contains px-captcha, the string _px, and the prompt “Activate and hold the button to confirm that you’re human.” That is a HUMAN Security (formerly PerimeterX) press-and-hold challenge. I did not find a cookie-less requests path that returns the real page, so I am not pasting any parsed product fields. There were none to paste.

How Walmart blocks you

Walmart fronts the site with HUMAN Security’s bot defense, which scores the whole request and serves a challenge when it is unsure. The block stacks several signals:

• TLS and HTTP/2 fingerprinting. A real Chrome connection has a specific TLS handshake (JA3/JA4) and HTTP/2 frame order. Python’s requests produces a different fingerprint, so the request looks non-browser before a single header is read.
• IP reputation. Datacenter and cloud IPs (where your script runs) are scored high-risk and pushed to the CAPTCHA.
• The press-and-hold challenge. The px-captcha widget needs a browser to run its JavaScript and register the hold gesture. A plain requests call has no browser, so it is stuck on the challenge with no way forward.
• The 200 status itself. Returning 200 instead of 403 is deliberate. It makes naive scrapers believe they succeeded and quietly collect challenge pages instead of data.

Matching the User-Agent solves none of these. To get past them with DIY tooling you would need a real browser engine (Playwright) plus rotating residential IPs and a way to clear the press-and-hold challenge, which is most of what a scraper API already bundles. Before going down the proxy rabbit hole, see how to scrape without getting blocked.

What about Playwright?

A headful browser can sometimes pass the challenge, and it is not a reliable fix on Walmart. Playwright drives a real browser engine, so it runs the JavaScript and can render the hydrated price. The remaining problems: the press-and-hold challenge still fires on flagged IPs, datacenter IPs stay high-risk no matter the engine, and HUMAN detects automation signals (WebDriver flags, timing) beyond the fingerprint. You would still need residential IPs and challenge handling on top of the browser. For a few pages from a clean residential IP, Playwright may get through; for volume, it stalls on the same wall. The Python web scraping guide covers the browser-automation setup if you want to try.

Method 2: scrape Walmart with a scraper API

When you need real product and price data from Walmart at any volume, a scraper API gets past the CAPTCHA that stopped Method 1. It runs the request through a real browser engine on a rotating residential IP and clears the challenge, so Walmart sees a browser-shaped session and returns the actual page.

ChocoData is the service I use for this category. It exposes a universal endpoint (one URL for any site) plus 453 dedicated endpoints for common targets. You hand it the Walmart URL and it returns the rendered page. Here is the request shape, swapping the raw requests.get from Method 1 for the API:

import requests

API_KEY = "cd_live_YOUR_KEY"  # free tier: 1,000 requests/month, no card
target = "https://www.walmart.com/ip/356659277"  # /ip/{itemId} resolves on the server

endpoint = "https://api.chocodata.com/api/v1/universal/get"
params = {
    "api_key": API_KEY,
    "url": target,
    "render": "true",   # run a real browser engine to clear the px-captcha
    "country": "us",     # US residential IP so price and stock resolve
}

resp = requests.get(endpoint, params=params, timeout=90)
print("STATUS:", resp.status_code)
# The rendered page (and embedded JSON) comes back in the payload. Parse the
# __NEXT_DATA__ JSON for title/price/availability, the same fields Method 1
# never reached because it got the "Robot or human?" page instead.

The difference from Method 1 is what happens behind that one call. ChocoData fetches through a country-matched residential IP that rotates automatically and retries through a fresh IP when a block appears, and the render flag runs a browser engine so the request is browser-shaped instead of the bare requests fingerprint Walmart challenged. The country=us parameter gives Walmart a US context so the price and stock resolve. Per ChocoData’s docs, billing applies only to successful (2xx) responses, so the challenge-page problem from Method 1 does not quietly drain your budget.

I did not paste live API output here because it requires a paid key, and I will not fake numbers. The endpoint path, parameters, free-tier limits, and billing model above come from ChocoData’s docs. The honest claim is narrow: the universal endpoint takes a URL and returns the rendered page, which removes the specific failure (the press-and-hold CAPTCHA) I reproduced in Method 1. I have not independently benchmarked their success rate on Walmart, so treat the parsing claims as vendor-stated until you test on the free tier.

Should you scrape Walmart yourself or use an API?

Use a scraper API. The DIY route does not return Walmart data in 2026, because the request never clears the CAPTCHA. The decision table is short:

My honest take from the test: a plain Python scraper does not work on Walmart, full stop, because the 200 it gets back is a “Robot or human?” challenge with no product data in it. Walmart has no public product API to fall back to the way eBay does, so the practical route to real data is a scraper API that renders the page and clears the CAPTCHA. Build the DIY version anyway to understand the block; just verify the response body, not the status code, so a 200 full of CAPTCHA never slips into your dataset.

To go deeper on the building blocks, see the Python web scraping guide for the language fundamentals, BeautifulSoup for parsing, and Scrapy if you are crawling many pages and want a framework. The web scraping pillar ties the whole workflow together.