Web Scraping with AI: LLM Extraction with LangChain

I have scraped sites with regex, with XPath, with Beautiful Soup, and with Scrapy. All of them share one weakness: the moment a site changes its markup, my selectors break and I am back in the HTML editing strings. AI scraping moves the brittle part to the model. You fetch the page, hand the HTML to an LLM with a description of the fields you want, and the model returns them as JSON. This guide builds that pipeline with LangChain and Pydantic, points the examples at books.toscrape.com (a sandbox built for practice), and is honest about where the approach costs you. The Python runtime is not installed on the machine I wrote this on, so the code below is the documented approach using each library’s current API, not a paste of command output I ran.

What is AI web scraping?

AI web scraping is data extraction where a large language model reads the page content and returns the fields you asked for, instead of you writing selectors that target specific HTML elements. You still fetch the page over HTTP. The difference is the parsing step: rather than soup.select("h1.product-title"), you give the model the HTML and a schema (“I want a title, a price as a float, and a stock count as an integer”), and it fills in the values.

The practical payoff shows up when sites change. A CSS selector is a hard-coded path into a specific DOM structure. Change the class name, wrap the element in a new div, or move the price into a different block, and the selector returns nothing. An LLM reads the page the way a person would, so it still finds the price after a redesign. The cost is that every page now runs through a model: you pay tokens, you wait for inference, and you have to guard against the model inventing a value.

Three pieces make up a working AI scraping pipeline:

The fetcher is the same problem it has always been, and the hardest one to keep working at scale (covered at the end). The schema and extractor are what “AI scraping” adds.

When should you use an LLM instead of CSS selectors?

Use an LLM when the page structure varies or changes often, and the field meaning stays stable. Use selectors when the structure is fixed and you scrape it at high volume. The decision is mostly about cost per page against tolerance for breakage.

A few rules I follow:

• Scraping 100,000 identical product pages from one site? Write selectors. The per-page model cost adds up fast, and a single layout means a single selector set.
• Scraping 200 competitor sites that all describe products differently? Use an LLM. Writing and maintaining 200 selector sets is the expensive path here.
• Scraping a site that redesigns every quarter? Use an LLM, or accept that you will rewrite selectors every quarter.
• Mix them. Many production scrapers use selectors for the stable bulk and fall back to an LLM only when a selector returns empty.

How do you install the AI scraping stack?

Install LangChain, the Anthropic integration, Beautiful Soup (used by the loader), and Pydantic. The examples use Anthropic’s Claude as the model, but LangChain’s init_chat_model lets you swap providers by changing one string.

pip install langchain langchain-community langchain-anthropic beautifulsoup4 pydantic

Set your model provider’s API key as an environment variable. LangChain reads it automatically, so it never appears in your code:

export ANTHROPIC_API_KEY="sk-ant-..."

Versions matter less than they used to here, because the two calls this tutorial leans on (.with_structured_output() and init_chat_model) have been stable in LangChain for a long time. The provider package (langchain-anthropic) tracks the model API, so keep it current.

How do you fetch a page with LangChain?

Use WebBaseLoader to pull a URL down to text. It wraps requests for the fetch and Beautiful Soup for the parse, and returns LangChain Document objects with the page content in .page_content.

from langchain_community.document_loaders import WebBaseLoader

loader = WebBaseLoader(web_paths=["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"])
docs = loader.load()

page_text = docs[0].page_content
print(len(page_text), "characters")

WebBaseLoader accepts a few arguments worth knowing:

• web_paths: a list of URLs to fetch. Pass several and .load() returns one Document per URL.
• header_template: a dict of HTTP headers. Set a real User-Agent here, because the library’s default identifies itself as Python and many sites reject that. See user agents for scraping for what to send.
• requests_per_second: throttle when loading many URLs, so you do not hammer the target.
• bs_kwargs: options passed through to Beautiful Soup, for example {"parse_only": ...} with a SoupStrainer to keep only part of the page.

For one-off fetches .load() is fine. For many URLs, .aload() runs them concurrently and .lazy_load() yields documents one at a time so you do not hold everything in memory.

One thing to note: WebBaseLoader fetches static HTML. It does not run JavaScript, so content rendered client-side will be missing, exactly like Beautiful Soup on its own. For JS-heavy pages you fetch with a browser engine or a scraper API first, then feed that HTML into the extraction step below.

How do you define the data schema with Pydantic?

Define a Pydantic BaseModel where each field is the data you want and the description tells the model what to look for. The class doubles as your extraction spec and your validator: the model is forced to return JSON matching it, and Pydantic rejects anything that does not fit the types.

from pydantic import BaseModel, Field

class Book(BaseModel):
    """A single book listing from a bookstore product page."""
    title: str = Field(description="The book's title")
    price_gbp: float = Field(description="Price in GBP as a number, without the currency symbol")
    in_stock: bool = Field(description="True if the page says the book is available")
    stock_count: int | None = Field(
        default=None,
        description="Number of units in stock if stated, otherwise null",
    )
    description: str | None = Field(
        default=None,
        description="The product description paragraph if present",
    )

Two habits make extraction far more reliable:

• Write descriptions for every field. The model reads them. “Price in GBP as a number, without the currency symbol” gets you 51.77; a bare price: float sometimes gets you a string with a pound sign that fails validation.
• Make optional fields actually optional. Mark anything that might be absent as | None with a default. Otherwise the model feels pressure to fill a required field and may invent a value, which is the main way LLM scraping goes wrong.

How do you extract structured data with an LLM?

Bind the schema to a chat model with .with_structured_output(), then invoke it with the page text. The method constrains the model so its reply parses into your Pydantic class, and .invoke() returns an instance of that class, not a raw string you have to json.loads().

from langchain.chat_models import init_chat_model
from langchain_community.document_loaders import WebBaseLoader
from pydantic import BaseModel, Field

class Book(BaseModel):
    """A single book listing from a bookstore product page."""
    title: str = Field(description="The book's title")
    price_gbp: float = Field(description="Price in GBP as a number, without the currency symbol")
    in_stock: bool = Field(description="True if the page says the book is available")
    stock_count: int | None = Field(default=None, description="Units in stock if stated, else null")
    description: str | None = Field(default=None, description="Product description paragraph if present")

# 1. Fetch
url = "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
page_text = WebBaseLoader(web_paths=[url]).load()[0].page_content

# 2. Pick a model. Swap the string to change providers.
model = init_chat_model("claude-haiku-4-5", model_provider="anthropic")

# 3. Bind the schema and extract
extractor = model.with_structured_output(Book)
book = extractor.invoke(
    f"Extract the book details from this page content:\n\n{page_text}"
)

print(book.title)        # "A Light in the Attic"
print(book.price_gbp)    # 51.77
print(book.in_stock)     # True
print(type(book))        # <class '__main__.Book'>

The values in the comments above are what this page returns when the pipeline runs correctly. I have not pasted real execution output because the Python runtime was not available where I wrote this, so treat those as the expected shape, not a logged result. The structure of the call is the documented LangChain API.

A note on model choice. init_chat_model("claude-haiku-4-5", ...) picks a fast, low-cost model, which is the right default for extraction: pulling named fields out of text is not a reasoning-heavy task, so the cheapest capable model usually does fine. Step up to a larger model only if a small one misreads your specific pages. The current model lineup and per-token prices:

Prices are from Anthropic’s published rates as of June 2026. A typical stripped product page runs a few thousand input tokens, so Haiku-class extraction lands around half a cent per page.

How do you extract many items from one page?

Wrap a list of your item schema in a parent model and extract that. The same .with_structured_output() call handles a listing page that contains many products, not just a single detail page.

from typing import List
from pydantic import BaseModel, Field

class BookListItem(BaseModel):
    title: str = Field(description="The book's title")
    price_gbp: float = Field(description="Price in GBP as a number")

class BookList(BaseModel):
    """All book listings found on a catalogue page."""
    books: List[BookListItem] = Field(description="Every book shown on the page")

url = "https://books.toscrape.com/catalogue/page-1.html"
page_text = WebBaseLoader(web_paths=[url]).load()[0].page_content

result = init_chat_model("claude-haiku-4-5", model_provider="anthropic") \
    .with_structured_output(BookList) \
    .invoke(f"Extract every book listing from this page:\n\n{page_text}")

for book in result.books:
    print(book.title, book.price_gbp)

This works well up to a point. A catalogue page with 20 items is fine. A page with 500 rows pushes both the input HTML and the output JSON up, raising cost and the chance the model drops or duplicates a row. Past a few dozen items, selectors are usually the better tool for the list extraction, with the LLM reserved for the detail pages. Know your page sizes before you decide.

How do you cut token cost before calling the model?

Strip the HTML down to the part that holds your data before you send it. Raw page HTML is mostly <script>, <style>, navigation, and footer markup that the model does not need, and you pay input tokens for every byte of it.

The cheapest win is using WebBaseLoader’s Beautiful Soup pass-through to keep only the relevant block. A SoupStrainer parses just the elements you name:

from bs4 import SoupStrainer
from langchain_community.document_loaders import WebBaseLoader

# Keep only the product description region on books.toscrape.com
only_main = SoupStrainer("article", class_="product_page")

loader = WebBaseLoader(
    web_paths=["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"],
    bs_kwargs={"parse_only": only_main},
)
page_text = loader.load()[0].page_content   # much smaller than the full page

Other reductions, in rough order of payoff:

• Drop scripts and styles. On a typical product page this alone removes a large share of the bytes. WebBaseLoader already strips tags to text, which handles most of it.
• Target a container. A SoupStrainer on the main content block, as above, often cuts another 50%+ versus the whole page.
• Truncate sensibly. If you only need fields from the top of the page, you do not need to send the comments section. Slice page_text before the call.
• Cache the schema. Reusing the same Pydantic schema across many calls lets the model provider cache the compiled output format, so repeated requests skip part of the setup cost.

Smaller input is not only cheaper. It is more accurate, because the model has less irrelevant text to confuse the field it is looking for.

How do you run AI scraping at scale without getting blocked?

The extraction code does not change at scale. What breaks is the fetch: the same anti-bot defenses that block any scraper block this one, and the model never sees a page you could not download. This is the part that decides whether your AI pipeline produces data or a folder of 403s.

Two failure modes dominate once you go past a few hundred pages:

• Blocks and CAPTCHAs. Datacenter IPs get rate-limited or banned, and challenge pages return HTML that contains a CAPTCHA instead of your data. Feed that to the model and it dutifully extracts fields from the CAPTCHA page. The fixes are residential IP rotation, real browser headers, and pacing. See scraping without getting blocked for the full playbook.
• JavaScript rendering. WebBaseLoader fetches static HTML, so any content injected by client-side JavaScript is simply absent. You need a headless browser to render the page first, then extract from the rendered DOM.

You can build both yourself: maintain a proxy pool, run and rotate browser fingerprints, solve challenges, and keep it all alive as targets change their defenses. Most teams find that becomes a full-time job that has nothing to do with the data they actually wanted.

The alternative is to put a scraper API in front of the fetch step and keep your LangChain extraction unchanged. ChocoData is one such option: it exposes a universal endpoint (/api/v1/{site}/{resource}) plus 453 dedicated endpoints, handles proxy rotation, browser rendering, and CAPTCHA solving on its side, and bills only on successful responses. You call it for the HTML, then hand that HTML to .with_structured_output() exactly as in the examples above. The model still does the extraction; the API just makes sure there is a real page to extract from.

import requests
from langchain.chat_models import init_chat_model

# Fetch through a scraper API (renders JS, rotates IPs server-side)
resp = requests.get(
    "https://chocodata.com/api/v1/universal",
    params={
        "api_key": "YOUR_KEY",
        "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
        "render_js": "true",
    },
    timeout=60,
)
resp.raise_for_status()
page_html = resp.text

# Same extraction as before, now on HTML that actually loaded
book = init_chat_model("claude-haiku-4-5", model_provider="anthropic") \
    .with_structured_output(Book) \
    .invoke(f"Extract the book details:\n\n{page_html}")

That split is the whole point: the scraper API owns the unreliable network problem, the LLM owns the brittle parsing problem, and you maintain neither a proxy pool nor a wall of CSS selectors.

Where to go next

AI extraction is one layer on top of the same scraping fundamentals. If you are new to the underlying mechanics, start with the web scraping with Python guide and the Beautiful Soup guide, since WebBaseLoader is built on requests and Beautiful Soup. For the broader landscape of tools, techniques, and legal considerations, the web scraping pillar is the map. And before you scale any of this, read scraping without getting blocked, because the model is only as good as the pages you manage to fetch.