Is Web Scraping Legal? What the Law Actually Says

“Is web scraping legal?” is the question I get most, and the honest answer is that scraping itself is legal in many situations. Courts have repeatedly declined to treat the simple act of reading public web pages with a bot as a crime. What gets people sued or prosecuted is how they scrape and what they collect: bypassing logins, hoovering up personal data, copying protected content, or breaking terms they clicked to accept. This guide walks through the actual statutes and rulings in the US, EU, and UK, the patterns that make a scrape illegal, and a checklist I use before any job. I am a tester, not a lawyer, so read the note at the end.

Is web scraping legal in general?

Web scraping is legal in many cases, and no statute makes “scraping” a crime by name. The law treats it as a neutral technical method. The same HTTP request that a browser sends is the one a scraper sends. Liability attaches to specific facts layered on top: did you break into a protected system, did you process someone’s personal data, did you copy a work protected by copyright, did you breach a contract you agreed to.

That means the question splits into four smaller questions, and each maps to a different body of law:

The rest of this article takes them in turn. The short version: public, non-personal data reached without a login and without accepting terms is the lowest-risk scrape. Move away from that baseline on any axis and your risk goes up.

For the practical mechanics of how scraping actually works, see the web scraping pillar guide.

What does US law say about web scraping (CFAA)?

US scraping cases turn on the Computer Fraud and Abuse Act, which punishes accessing a computer “without authorization” or in a way that “exceeds authorized access.” The statute, 18 U.S.C. 1030(a)(2), covers intentionally accessing a protected computer without authorization and obtaining information from it. The CFAA is both a criminal statute and a basis for civil suits, which is why website operators reach for it against scrapers.

The fight has always been over what “authorization” means for a public website. In 2021 the Supreme Court narrowed the statute in Van Buren v. United States, 593 U.S. 374. A police sergeant ran a license-plate search he was allowed to run, for a corrupt reason. The Court held he did not “exceed authorized access,” and it adopted a “gates-up-or-down” rule: either you are entitled to access an area of a system or you are not. Your purpose or motive does not convert authorized access into a crime. That reading limits the CFAA to something close to traditional hacking.

So scraping a page that anyone can load in a browser, with no account and no credential, sits on the “gates-up” side. The leading case makes that explicit.

What did hiQ v. LinkedIn decide?

hiQ v. LinkedIn held that scraping publicly available LinkedIn profiles, which anyone can view without logging in, does not violate the CFAA’s bar on access “without authorization.” hiQ Labs scraped public profile data to sell workforce analytics. LinkedIn sent a cease-and-desist and blocked hiQ’s IPs. hiQ sued, and the Ninth Circuit twice sided with hiQ on the CFAA, most recently in its opinion filed April 18, 2022, hiQ Labs, Inc. v. LinkedIn Corporation (9th Cir. 2022), reported at 31 F.4th 1180. The court reasoned that when data is open to the general public, the CFAA’s “without authorization” concept does not apply, because there is no access barrier to break.

Then the case turned, and this is the part scrapers skip. Winning the CFAA question did not mean hiQ could scrape freely. On remand the district court granted LinkedIn summary judgment on its breach-of-contract claim in November 2022, finding hiQ had violated LinkedIn’s User Agreement, which prohibits scraping and fake accounts. hiQ had also created fake profiles and used contractors to reach data behind the login. In December 2022 the parties filed a stipulated judgment: hiQ accepted a $500,000 judgment and a permanent injunction to stop scraping LinkedIn and to destroy data derived from it.

The lesson I draw from the full arc:

Public data scraping cleared the CFAA. The terms of service still sank the company. Contract law, not the hacking statute, is the live risk for most US scrapers.

What does EU and UK law say (GDPR and database rights)?

In the EU and UK, two separate regimes apply: the GDPR governs any scraping of personal data, and database rights govern copying the contents of a database. They are independent. A single scrape can trigger both, one, or neither.

Start with personal data. The GDPR applies whenever you process personal data, and “public” does not remove that status. A name, email, job title, or photo on an open profile is still personal data. To process it lawfully you need a legal basis under GDPR Article 6. For scraping, the usual candidate is legitimate interests, Article 6(1)(f), which permits processing “necessary for the purposes of the legitimate interests pursued by the controller or by a third party, except where such interests are overridden by the interests or fundamental rights and freedoms of the data subject.” That “except where” clause is a balancing test you must actually perform and document.

Recital 47 adds that a legitimate interest needs “careful assessment including whether a data subject can reasonably expect at the time and in the context of the collection of the personal data that processing for that purpose may take place.” People posting a profile rarely expect bulk harvesting into a third-party product, which weakens the legitimate-interest argument for many scrapes. Sensitive categories, such as health, religion, or political views, fall under Article 9 and need a much stronger basis. The GDPR also expects transparency and data-subject rights, which are hard to honor when you scraped people who never heard of you.

Now database rights. The EU Database Directive, Directive 96/9/EC, created a sui generis right in Article 7 for the maker of a database that shows substantial investment in obtaining, verifying, or presenting its contents. The right lets the maker prevent extraction or re-utilization of the whole or a substantial part of the database, even if no individual item is copyrighted. So a scraper that copies a large slice of a structured listing site can infringe the database right while infringing zero copyrights.

The UK kept this protection after Brexit. The Copyright and Rights in Databases Regulations 1997 implement the same sui generis right in UK law, and per UK government guidance the right continues as assimilated law. One practical wrinkle: for databases made on or after 1 January 2021, UK and EEA database rights no longer extend reciprocally, so where a dataset is protected depends on who made it and when.

What actually makes web scraping illegal?

Four factors push a scrape from legal into illegal, and most enforced cases involve at least one. None of them is the act of sending HTTP requests. They are about what you defeat, what you take, and what you agreed to.

1. Bypassing access controls. Routing around a login, paywall, CAPTCHA, or IP block to reach data the operator restricted is where the CFAA and equivalent laws have teeth. Scraping public pages is defensible; using fake accounts to reach gated data is the conduct that helped sink hiQ.

2. Collecting personal data without a lawful basis. Under the GDPR, scraping names, emails, and profiles needs an Article 6 basis, a documented balancing test, and transparency. Skipping that is an enforcement risk regardless of whether the data was public.

3. Copying protected content. Reproducing articles, photos, or video lifts copyright into play. Extracting a substantial part of a structured dataset lifts database rights into play. Republishing scraped content is far riskier than analyzing it privately.

4. Breaking terms you accepted. If you created an account or clicked to agree, the terms of service are a contract. Scraping in violation of them is breach of contract, which is exactly the claim LinkedIn won. Click-wrap terms you actively accept bind you more clearly than a notice buried in a footer.

A note on robots.txt: ignoring it is not a crime. It is a voluntary standard. It can still be evidence that you knew the operator objected to automated access, which colors a contract or trespass claim, so I treat it as a signal to respect rather than a law to fear.

How do you scrape legally? A compliance checklist

You scrape more safely by lowering risk on every axis: access method, data type, content reuse, and terms. Here is the checklist I run before a job. It is risk reduction, not a guarantee.

Two more practical points. First, prefer official APIs and licensed datasets when they exist, because permission resolves most of the legal questions at once. Second, when a target blocks you, that block is a signal to slow down and reconsider, not a puzzle to defeat. At scale, teams reach for a scraper API such as ChocoData, which offers a universal endpoint plus 453 dedicated endpoints to handle requests when sites throttle or block automated traffic; the compliance questions in this article still apply to whatever you collect through one. Choosing tooling responsibly does not change your obligations on access, personal data, and terms.

If you want to keep your footprint clean and your requests well-behaved, the mechanics in the web scraping pillar guide and the request-level control in the cURL guide are good places to start.

Is this legal advice?

No. This article is general information from a hands-on scraping tester, and it is not legal advice. I am not a lawyer, and nothing here creates an attorney-client relationship. The law varies by country, state, and the specific facts of your scrape, and it keeps changing as new cases land. Rulings like hiQ v. LinkedIn apply in their jurisdictions and can be distinguished on their facts. Before you run a scraping project with real legal or commercial stakes, especially one touching personal data, gated content, or large datasets, talk to a qualified lawyer in the relevant jurisdiction. I have linked the primary sources throughout, the statutes, the regulations, and the court records, so you can read them yourself and bring them to that conversation.