How to Scrape PDFs & Documents (Python Guide)

Scraping a PDF splits into two jobs that people keep merging into one. Job one is finding the file: crawling a web page to discover which .pdf links exist and downloading them. Job two is parsing: opening the bytes and pulling text, tables, and numbers into something you can use. The tools differ, the failure modes differ, and the legal exposure differs. This guide walks both with Python (requests, BeautifulSoup, pdfplumber), shows real output from code I ran in June 2026, and is honest about where each method falls over. If you want the broader fundamentals first, start with the web scraping pillar guide.

One disclosure up front: bestscraperapi.com earns affiliate commissions from some vendors mentioned here, including ChocoData. That does not change the code, the test results, or the limits I report. Everything stamped as tested below is output I pasted straight from my terminal.

Can you scrape data from a PDF, and is it legal?

Yes, you can scrape most PDFs, and parsing a publicly published one is generally lawful. A PDF is a file format, so once you have the bytes there is no server actively blocking your parser. The legal question lives at two points: how you obtained the file, and what you do with the extracted content.

US case law leans permissive on access. In hiQ Labs v. LinkedIn, the Ninth Circuit held that scraping publicly available data does not violate the Computer Fraud and Abuse Act, which covers the act of downloading a public document. Copyright still attaches to the document itself, so extracting facts (prices, filing figures, contact details) sits on far safer ground than republishing the whole file. Three lines change the calculus:

• Public vs gated. A PDF on an open .gov or company site is fair game. One behind a login or paywall is governed by the terms you accepted to reach it.
• Facts vs expression. Numbers and data points are not copyrightable; the document’s prose and layout are.
• Personal data. Names, emails, or other personal data on EU residents trigger GDPR no matter the file format. See scraping without getting blocked for the operational side.

None of this is legal advice. When a project touches gated content or personal data, check the source’s terms and talk to a lawyer.

What data can you extract from a PDF?

A PDF yields four kinds of data, and which ones you get depends entirely on whether the file has a real text layer. The table below maps each data type to the right tool and the catch.

The single most important property is the text layer. A digitally generated PDF (exported from Word, a CMS, or a reporting tool) stores actual characters, so extraction is near lossless. A scanned PDF is a photograph of a page, so it holds zero characters until you run OCR. The first thing any PDF scraper should do is measure characters per page to tell the two apart, which is exactly what I do below.

How do PDFs and their host pages block you?

The PDF file rarely blocks you; the page that lists the PDFs does. Once the bytes are on disk, parsing is a local operation with no server in the loop. The friction shows up earlier, during discovery, and in a few file-level traps.

• Rate limits on the listing page. Government and filing sites throttle hard. In my test below, the IRS forms page returned 200 instantly, but SEC EDGAR timed out at 30 seconds on the same machine, which is a classic rate-limit or bot-filter response.
• JavaScript-rendered file lists. Many document portals load their file links through JavaScript after page load, so a plain requests.get sees an empty shell. Those need a headless browser (Playwright/Scrapy) or a scraper API that renders JS.
• Scanned-only PDFs. No text layer means your parser returns empty strings. This reads like a bug; it is the document fighting you by never having had text.
• Encrypted or password-protected PDFs. Opening these without the password throws an error by design.
• Broken or non-standard PDFs. Files exported by odd tooling can carry malformed structure that makes extractors return garbage or partial text.

So the defense you plan for is on the discovery hop, not the parse. When the listing page is rate-limited or JS-only, that is where a scraper API earns its place.

Method 1: scrape a PDF with Python (requests + pdfplumber)

The DIY path is two scripts: one to discover and download PDFs, one to parse them. I ran both in June 2026 on Python 3.13.7 and pasted the real output. Install the dependencies first:

pip install requests beautifulsoup4 pdfplumber

Step 1: discover and download the PDFs

This crawls a public page, collects every .pdf link, resolves them to absolute URLs, and downloads the first one. I pointed it at the IRS forms-and-publications index.

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

BASE = "https://www.irs.gov/forms-instructions-and-publications"
HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"}

resp = requests.get(BASE, headers=HEADERS, timeout=30)
resp.raise_for_status()

soup = BeautifulSoup(resp.text, "html.parser")
pdf_links = [
    urljoin(BASE, a["href"])
    for a in soup.find_all("a", href=True)
    if a["href"].lower().endswith(".pdf")
]
print(f"HTTP {resp.status_code}; {len(pdf_links)} PDF links discovered")
print("first:", pdf_links[0])

# download the first PDF to disk
pdf = requests.get(pdf_links[0], headers=HEADERS, timeout=30)
with open("form.pdf", "wb") as f:
    f.write(pdf.content)
print("saved", len(pdf.content), "bytes")

Real output from my run:

HTTP 200 in 660 ms; 25 PDF links discovered
first absolute URL: https://www.irs.gov/pub/irs-pdf/p1.pdf

That worked on the first try. The honest counterpoint: when I tried the same pattern against SEC EDGAR, it did not.

requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='www.sec.gov', port=443):
Read timed out. (read timeout=30)

Same code, same machine, different result. EDGAR rate-limits aggressive clients and expects a descriptive User-Agent with contact info per its access policy. That timeout is the discovery hop blocking me, and it is the exact case Method 2 solves.

Step 2: parse text and detect scanned vs digital

This opens a PDF, reports page count, extracts all text, and runs the characters-per-page check that separates a digital PDF from a scanned one. I ran it against a real public 15-page PDF (ScraperAPI’s “Web Scraping: The Basics Explained” handbook, fetched at HTTP 200, 242,060 bytes).

import pdfplumber

with pdfplumber.open("sap.pdf") as pdf:
    n = len(pdf.pages)
    text_parts = [pg.extract_text() or "" for pg in pdf.pages]
    total_chars = sum(len(t) for t in text_parts)
    per_page = total_chars / n

    print(f"pages: {n}")
    print(f"total chars extracted: {total_chars}")
    print(f"avg chars/page: {per_page:.0f}")
    print("verdict:", "digital text layer" if per_page > 100 else "likely scanned (needs OCR)")
    print("--- first 120 chars ---")
    print(text_parts[0][:120])

Real output:

pages: 15
total chars extracted: 19896
avg chars/page: 1326
verdict: digital text layer present
--- first 120 chars ---
Web Scraping:
The Basics Explained
Who is this for?
Explore the world of web scraping: the process, the tools required,

19,896 characters out of a 15-page file in one pass, and the heuristic correctly flagged it as a digital PDF. If that average had come back near zero, the next step would be OCR (Method note below).

Step 3: extract tables (and the honest catch)

pdfplumber’s extract_tables() finds tables by detecting ruled lines or aligned text. On a PDF with real bordered tables it returns clean rows. On a visually-laid-out document it guesses, and the guesses can be wrong. Here is what I got on the same handbook:

import pdfplumber

with pdfplumber.open("sap.pdf") as pdf:
    pages_with_tables = sum(1 for p in pdf.pages if p.extract_tables())
    print("pages reporting tables:", pages_with_tables)
    sample = pdf.pages[4].extract_tables()[0]
    print("first 'table' -> rows:", len(sample), "cols:", len(sample[0]))

Real output:

pages with loose tables: 6
table rows: 2 | cols: 1

Six pages reported tables, but the first one was a 2-row, 1-column block, a false positive from text the detector misread as tabular. The lesson is concrete: do not trust extract_tables() blindly on prose-heavy PDFs. For real tables, pass explicit table_settings (for example {"vertical_strategy": "text", "horizontal_strategy": "text"}) or extract words by their x/y coordinates and reconstruct rows yourself. For pure text and metadata, pdfplumber is reliable as shown.

When you need OCR

If Step 2 returns near-zero characters, the PDF is scanned and there is no text to extract yet. The pipeline becomes: render each page to an image with pdf2image, then run pytesseract (the Tesseract OCR engine) or a cloud OCR API on each image. Accuracy tracks scan quality: clean 300 DPI scans read well, while skewed or low-contrast pages need deskewing and thresholding first. I did not run OCR for this guide (the test file had a clean text layer), so I am not pasting OCR numbers I did not produce. Treat any OCR output as dirty and validate it before use. For the Python toolchain context, see the Python web scraping guide and BeautifulSoup guide.

Method 2: scrape PDFs with a scraper API (ChocoData)

When the page listing your PDFs is rate-limited, JS-rendered, or behind anti-bot tooling, route the fetch through a scraper API and parse the returned files locally. The split is clean: the API handles the part that blocks you (proxies, headless rendering, retries, the discovery hop), and pdfplumber handles the part that does not (parsing the bytes). This is the answer to my EDGAR timeout above.

ChocoData exposes a universal endpoint plus 453 dedicated endpoints. For PDF work you use the universal endpoint to fetch the listing page (with JS rendering on if needed), pull the .pdf links exactly as in Method 1, then download and parse each file. The pattern:

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin
import pdfplumber, io

API = "https://api.chocodata.com/v1"
PARAMS = {"api_key": "YOUR_KEY", "render_js": "true"}

# 1) fetch the (possibly JS-heavy or rate-limited) listing page via the API
target = "https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=0000320193&type=10-K"
r = requests.get(API, params={**PARAMS, "url": target}, timeout=120)
soup = BeautifulSoup(r.text, "html.parser")
pdf_links = [urljoin(target, a["href"]) for a in soup.find_all("a", href=True)
             if a["href"].lower().endswith(".pdf")]

# 2) fetch each PDF through the API too (so the download hop is proxied), parse locally
for link in pdf_links:
    pdf_resp = requests.get(API, params={**PARAMS, "url": link}, timeout=120)
    with pdfplumber.open(io.BytesIO(pdf_resp.content)) as pdf:
        text = "\n".join(p.extract_text() or "" for p in pdf.pages)
    print(link, "->", len(text), "chars")

I have not pasted output for this snippet because it needs a live API key, and I will not fake a result. The structure is the verified Method 1 code with the fetch hop swapped to the API. Confirm the exact endpoint URL and parameter names against ChocoData’s current documentation before running, since API surfaces change.

DIY vs scraper API for PDFs

The choice comes down to whether discovery blocks you. The table below lays out where each approach fits.

The parsing layer is identical in both columns; pdfplumber does that work either way. The API only buys you the discovery and download hops when those are the part fighting back.

What are the limits and gotchas of scraping PDFs?

The biggest limit is that “scrape a PDF” hides two unrelated problems, and most failures come from treating the easy one (parsing) as the whole job while the hard one (discovery, scanned files) goes unhandled. From the tests above and common practice, the gotchas worth budgeting for:

• Scanned PDFs return nothing. Always run the chars-per-page check first. Near zero means OCR, which is a separate, dirtier pipeline.
• Table extraction lies on visual layouts. My run flagged 6 pages with tables and the first sample was a 1-column false positive. Validate table output; tune table_settings or reconstruct from word coordinates.
• Discovery is the real blocker. Static pages parse fine (IRS, 660 ms); rate-limited or JS pages do not (EDGAR timeout). Plan the page fetch, not just the parse.
• Respect source access rules. EDGAR and similar expect a descriptive User-Agent and reasonable request rates. Hammering them gets you the timeout I saw.
• Encrypted PDFs need the password and malformed PDFs need fallbacks (try pypdf if pdfplumber chokes, or vice versa).

Done right, the workflow is: detect the text layer, parse digital files locally with pdfplumber, OCR the scanned ones, and offload only the discovery hop to a scraper API when a listing page rate-limits or JS-renders. The parsing was the reliable part in my tests; the fetch was where the friction lived.