How to Scrape Emails & Leads (Python Guide)

An email scraper extracts addresses out of the HTML of pages you point it at. It is the backbone of cold-outreach lead lists, recruiting pipelines, and partner research. This guide builds one in Python with requests and BeautifulSoup, shows the two extraction methods that actually hold up (a regex over the text, and parsing mailto: links), and is honest about the three things that break naive scrapers: obfuscation, JavaScript, and bot blocks. I ran both methods against live public pages in June 2026, and the output below is pasted verbatim.

Can you legally scrape emails and leads?

Collecting addresses from public pages is largely lawful in the US, and sending to them is governed separately. The split trips people up: getting the address and using the address answer to different laws.

On collection, the key precedent is hiQ Labs v. LinkedIn, where the Ninth Circuit held in 2022 that automated capture of data from publicly accessible pages (ones that need no login) does not violate the Computer Fraud and Abuse Act’s “without authorization” clause. The practical caveat: hiQ still lost on other grounds, settling and agreeing to destroy the scraped data, so breach-of-contract and other claims stay in play. If any address belongs to an EU resident, GDPR governs collection regardless of where you scrape: Article 6 requires a lawful basis, and Article 83 sets fines up to EUR 20 million or 4% of worldwide annual turnover.

On sending, the rules tighten by region. The table summarizes the lines for cold email:

Two specifics. The CAN-SPAM maximum is a per-email figure in the FTC’s own compliance guide, so a bad blast scales fast. And France draws a sharp B2C/B2B line: generic addresses (info@, contact@) need no consent at all, professional addresses can be emailed if the recipient can object, individuals must opt in first. None of this is legal advice; if you are scaling outreach, run your process past counsel. For the wider picture see my is web scraping legal overview, and check each address before you send, not after.

What lead data can you actually pull from a page?

Public pages expose more than the address, and a good scraper grabs the surrounding fields in the same pass. Here is what is realistically available and how reliably you get it:

The honest limit: a contact page usually hands you role inboxes (sales@, support@), not named decision-makers. Team and “about” pages are where personal addresses surface, and those are also where obfuscation is heaviest. Plan your target list accordingly.

How do pages block or hide their emails?

Most lead-bearing pages fight scrapers in one of three ways, and each defeats a naive regex. Knowing which one you are facing tells you which method to reach for.

The two methods below cover clean text and mailto: links, the cases you can solve with requests alone. Obfuscation, JavaScript, and IP blocks push you toward Playwright or a scraper API, which I cover at the end. Before scraping anything, read the site’s robots.txt and respect it. The W3C page I test below allows it: its robots.txt disallows only WordPress admin paths.

Method 1: How do you build an email scraper with Python and BeautifulSoup?

Fetch the page, run a regex for the email pattern over the parsed text, and dedupe. This is the fastest method and it works whenever the address is present as plain text in the HTML. I pointed it at the W3C contact page, a stable public page that lists its team inboxes. Here is the exact code I ran:

import re
import requests
from bs4 import BeautifulSoup

EMAIL_RE = re.compile(r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}")
HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; lead-research/1.0)"}

url = "https://www.w3.org/Consortium/contact"
resp = requests.get(url, headers=HEADERS, timeout=20)
print("STATUS:", resp.status_code)

soup = BeautifulSoup(resp.text, "html.parser")
text = soup.get_text(" ", strip=True)
emails = sorted(set(EMAIL_RE.findall(text)))

print("FOUND:", len(emails))
for e in emails:
    print("-", e)

The real output:

STATUS: 200
FOUND: 10
- contact@w3.org
- giving@w3.org
- invoicing@w3.org
- membership@w3.org
- site-comments@w3.org
- sysreq@w3.org
- team-china-contact@w3.org
- team-liaisons@w3.org
- team-wcap-contact@w3.org
- w3t-pr@w3.org

Three details make this robust. I run the regex over soup.get_text() rather than resp.text, which strips script and style blocks so I do not match addresses buried in tracking code. The set() dedupes, because the same inbox often appears in both a link and the footer. And I set a real User-Agent: many sites serve a stripped page or a block to the default python-requests agent.

The regex itself is deliberately simple and will catch obvious false positives like you@example.com in sample text, plus version strings that look address-like. For a lead list, filter the results: drop addresses whose domain does not match the site you are on, and discard known placeholder domains (example.com, sentry.io, your-domain). The pattern also misses obfuscated addresses on purpose, which the next method and the scraper-API section address.

Method 1b: How do you catch emails hidden in mailto: links?

Parse the href of every mailto: anchor instead of reading the visible text. Many sites wrap the address in a link and never print it as plain text, so a text regex alone returns nothing while the address sits right there in the markup. I ran this against the GNU contact page, which uses mailto: links throughout:

import requests
from urllib.parse import unquote
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; lead-research/1.0)"}
url = "https://www.gnu.org/contact/"
resp = requests.get(url, headers=HEADERS, timeout=20)
print("STATUS:", resp.status_code)

soup = BeautifulSoup(resp.text, "html.parser")
emails = []
for a in soup.select("a[href^=mailto]"):
    addr = unquote(a["href"][7:].split("?")[0])  # strip 'mailto:' and ?subject=
    if addr:
        emails.append(addr)

emails = sorted(set(emails))
print("FOUND:", len(emails))
for e in emails:
    print("-", e)

The real output:

STATUS: 200
FOUND: 8
- assign@gnu.org
- gnu@gnu.org
- license-violation@gnu.org
- licensing@gnu.org
- maintainers@gnu.org
- sysadmin@gnu.org
- web-translators@gnu.org
- webmasters@gnu.org

The slice a["href"][7:] drops the mailto: prefix, .split("?")[0] trims any ?subject= query that a link carries, and unquote decodes URL-encoded characters. In production, run both methods on every page and merge the results: the regex catches plain-text addresses, the mailto: parser catches linked ones, and the union is your row. If you want to learn the selector mechanics behind a[href^=mailto], my CSS selectors guide covers attribute matching in depth, and the BeautifulSoup guide covers parsing end to end.

How do you crawl multiple pages and dedupe a real lead list?

Maintain a set of seen addresses, loop your target URLs, run both extractors on each, and sleep between requests. A single contact page is a demo; a lead list means visiting many. Here is the pattern, combining both methods into one reusable function:

import re, time, requests
from urllib.parse import unquote
from bs4 import BeautifulSoup

EMAIL_RE = re.compile(r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}")
HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; lead-research/1.0)"}

def emails_from(url):
    r = requests.get(url, headers=HEADERS, timeout=20)
    if r.status_code != 200:
        return set()
    soup = BeautifulSoup(r.text, "html.parser")
    found = set(EMAIL_RE.findall(soup.get_text(" ", strip=True)))
    for a in soup.select("a[href^=mailto]"):
        found.add(unquote(a["href"][7:].split("?")[0]))
    return {e for e in found if e}

targets = [
    "https://www.w3.org/Consortium/contact",
    "https://www.gnu.org/contact/",
]

leads = set()
for url in targets:
    leads |= emails_from(url)
    time.sleep(2)          # be polite: one request every 2 seconds

print(f"{len(leads)} unique addresses across {len(targets)} pages")

Three production notes. The time.sleep(2) keeps you under most rate limits; tune it, but do not remove it. The if r.status_code != 200 guard means a single blocked page does not crash the run. And feeding the union of both extractors into one set gives you free deduplication across the whole crawl. For a larger, structured crawl with retries and queues, graduate to Scrapy, which handles concurrency and politeness for you.

Method 2: When should you use a scraper API instead?

Use a scraper API once your targets obfuscate addresses, render them with JavaScript, or block your IP after a few requests, because that is exactly when requests and BeautifulSoup stop returning data. The W3C and GNU pages above are well-behaved. Real lead targets (large directories, marketplaces, JS-heavy team pages, anything behind aggressive bot detection) are not. At scale your IP gets a 403, the address never appears in the raw HTML because a script injects it, or a CAPTCHA blocks the page entirely. For why this happens and the wider defenses, see scraping without getting blocked.

A scraper API solves the transport layer: it rotates residential proxies, clears CAPTCHAs, and renders JavaScript, handing you the final HTML so your extraction code stays simple. ChocoData exposes a universal endpoint that, per its own pages in mid-2026, works across 235 sites, plus 453 dedicated endpoints in total. You call the universal endpoint with a target URL, it returns rendered HTML, and you run the same two extractors from above against the response. Here is the shape (add your key to run it):

import re, requests
from bs4 import BeautifulSoup

EMAIL_RE = re.compile(r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}")

resp = requests.get(
    "https://api.chocodata.com/v1/universal",
    params={
        "api_key": "YOUR_KEY",
        "url": "https://example.com/team",
        "render_js": "true",          # execute JavaScript before returning HTML
    },
    timeout=60,
)
soup = BeautifulSoup(resp.text, "html.parser")
emails = sorted(set(EMAIL_RE.findall(soup.get_text(" ", strip=True))))
print(emails)

The extraction logic is identical to Method 1; the only change is that the API delivers HTML that did not block you and that already ran its scripts. On pricing, from ChocoData’s own pricing page in mid-2026:

One standard request is 5 credits; JavaScript rendering adds +10. Pay-as-you-go top-ups run $0.90 per 1,000 successful requests, and ChocoData states only successful (2xx) responses are billed, so block pages do not cost you. The free tier needs no card. ChocoData is a general scraper API, not an email database: it gives you the rendered HTML at scale, and you still parse and verify the addresses yourself with the code above.

Should you scrape pages or buy from a finder database?

Match the approach to your target. If your leads are businesses with a known name and domain, a finder database (Hunter, Apollo, Snov.io) returns a scored, often pre-verified address in one call and is cleaner than crawling. If your leads live on arbitrary sites no database has indexed (regional directories, niche marketplaces, freshly published pages), you scrape, with requests and BeautifulSoup for clean pages and a scraper API for blocked or JS-heavy ones. I compare the named finder and crawler tools, their prices, and free tiers in best email scrapers, so I will not re-rank them here.

Whichever path you take, the address is the cheap part. Scraping returns syntactically valid strings, not confirmed mailboxes, so budget real effort for verification (syntax, MX lookup, SMTP probe) and treat the send as a separate compliance project under CAN-SPAM and GDPR. Get the extraction right with the two methods above, verify before you send, and keep a documented lawful basis for any EU address. For the fundamentals behind all of it, headers, pagination, parsing, and etiquette, start with the web scraping pillar guide.