How to Scrape GitHub: API and HTML Methods (Tested)

GitHub holds the metadata for almost every open-source project: stars, languages, contributors, release history, issues. The good news is you rarely need to scrape its HTML. GitHub ships a generous, well-documented REST API that hands back clean JSON. This guide leads with that API, then shows a BeautifulSoup fallback for the handful of pages the API does not cover. Every snippet below ran against live GitHub data in June 2026, and the output is pasted verbatim.

Should you use the GitHub API or scrape the HTML?

Use the API. It is the right tool for almost every GitHub data task, and HTML scraping should be your fallback. The REST API returns structured JSON, is documented at docs.github.com/rest, and gives you 5,000 requests per hour once you add a free token. Scraping the rendered HTML is slower, breaks when GitHub ships a layout change, and pulls down megabytes of markup to find one number the API would hand you directly.

Here is how the two approaches compare for GitHub work specifically:

The takeaway is simple. Reach for HTML parsing only when the data lives on a page the API does not expose, such as the pinned repositories on a profile. Everything else goes through the API.

How do you list a user’s repositories with the GitHub API?

Send a GET request to https://api.github.com/users/{username}/repos and you get back a JSON array of that user’s public repositories. No authentication is required at low volume. Here is the exact code I ran against Linus Torvalds’ account, capped at five repos:

import requests

r = requests.get("https://api.github.com/users/torvalds/repos?per_page=5")
print("STATUS:", r.status_code)
print("X-RateLimit-Limit:", r.headers.get("X-RateLimit-Limit"))
print("X-RateLimit-Remaining:", r.headers.get("X-RateLimit-Remaining"))
print()
for repo in r.json():
    print(repo["name"], "-", repo["stargazers_count"], "stars -", repo["language"])

That returned the status, the rate-limit headers, and one line per repo:

STATUS: 200
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 59

1590A - 566 stars - OpenSCAD
AudioNoise - 4382 stars - C
GuitarPedal - 2003 stars - C
HunspellColorize - 346 stars - C
libdc-for-dirk - 389 stars - C

Two things to notice. The X-RateLimit-Limit header reads 60, which is the anonymous hourly cap, and X-RateLimit-Remaining ticks down with every call. Each repo object carries far more than I printed: forks_count, open_issues_count, created_at, default_branch, license, and the full clone URLs. Print repo.keys() once to see the whole shape.

To page past the first results, add &page=2 and keep incrementing until the array comes back empty. Sort with &sort=updated or &sort=pushed to get the most active repos first.

How do you search repositories with the GitHub API?

Hit the https://api.github.com/search/repositories endpoint with a q query parameter and GitHub returns the matching repositories ranked by relevance, plus a total_count of every match. This is the same engine behind GitHub’s own search box. Here is the code I ran for the query “web scraping”:

import requests

r = requests.get("https://api.github.com/search/repositories?q=web+scraping&per_page=3")
data = r.json()
print("STATUS:", r.status_code)
print("total_count:", data["total_count"])
print()
for repo in data["items"]:
    print(repo["full_name"], "-", repo["stargazers_count"], "stars")
    print("   ", repo["description"][:70])

The real output:

STATUS: 200
total_count: 131750

scrapy/scrapy - 62211 stars
    Scrapy, a fast high-level web crawling & scraping framework for Python
soimort/you-get - 56836 stars
    :arrow_double_down: Dumb downloader that scrapes the web
lorien/grab - 2460 stars
    Web Scraping Framework

GitHub found 131,750 repositories for that query and returned the top three by relevance. The search results live under data["items"], not at the top level, which trips people up the first time. You can sharpen queries with qualifiers: q=web+scraping+language:python+stars:>1000 filters to Python repos above a thousand stars. The full qualifier list is in the search syntax docs.

One catch worth flagging: the Search API has its own rate limit, separate from the core API. Anonymous search allows 10 requests per minute. A token raises that to 30 per minute.

How do GitHub API rate limits and authentication work?

Anonymous requests are limited to 60 per hour per IP address, and a personal access token raises that to 5,000 per hour. You saw the X-RateLimit-Limit: 60 header in the first test, which confirms the anonymous cap. For any real project, generate a token under Settings, Developer settings, Personal access tokens (a classic token with no scopes works for public read-only data), then send it in the Authorization header:

import requests

headers = {"Authorization": "Bearer YOUR_TOKEN_HERE"}
r = requests.get("https://api.github.com/users/torvalds/repos", headers=headers)
print(r.headers.get("X-RateLimit-Limit"))  # now 5000

Always read the rate-limit headers rather than guessing. X-RateLimit-Remaining tells you how many calls are left in the window, and X-RateLimit-Reset gives the Unix timestamp when the window refills. When Remaining hits 0, the API returns a 403 until the reset time. Polite clients check Remaining and back off before they get blocked. GitHub also asks that you set a User-Agent header identifying your app; requests sends a default one, but a descriptive value is better etiquette.

When should you scrape GitHub HTML instead of using the API?

Scrape the HTML only for data the API does not expose, and pinned repositories are the classic example. A user picks up to six repos to feature on their profile, and that selection is not available through any REST endpoint. For that, I fetch the profile page and parse it with BeautifulSoup. Here is the code I ran:

import requests
from bs4 import BeautifulSoup

UA = {"User-Agent": "Mozilla/5.0 (research)"}
html = requests.get("https://github.com/torvalds", headers=UA, timeout=30).text
soup = BeautifulSoup(html, "html.parser")

pinned = soup.select("span.repo")
print("Pinned repositories:", len(pinned))
for p in pinned:
    print("-", p.get_text(strip=True))

The live result:

Pinned repositories: 5
- linux
- GuitarPedal
- uemacs
- AudioNoise
- HunspellColorize

The span.repo selector grabs the repo-name element inside each pinned card. This works today, and it will break the day GitHub renames that CSS class, which is the whole problem with HTML scraping and the reason the API comes first. If you want to learn the selector mechanics behind this, my CSS selectors guide covers how to target elements reliably. Set a real User-Agent here too: GitHub serves a stripped page or a block to generic bot agents.

Stay inside GitHub’s rules when you parse HTML. The Acceptable Use Policies permit scraping public, non-personal information for research and archiving, and they prohibit scraping personal data for spam or for building a competing service. Pulling public repo metadata for analysis is fine. Harvesting user emails is not.

How do you scale GitHub scraping when you hit limits?

Add authentication first, then control your request rate against the headers. The single biggest lever is a token, which moves you from 60 to 5,000 requests per hour. Beyond that, cache responses you have already fetched, request only the pages you need with per_page and page, and sleep when X-RateLimit-Remaining runs low instead of slamming into a 403.

If you are scraping GitHub’s HTML pages at scale, the rendered profile and trending pages start returning blocks once a single IP makes too many requests in a short window. This is where the scraper-API category fits. A service like ChocoData, which offers a universal endpoint plus 453 dedicated endpoints, handles proxy rotation and retries so a large HTML crawl keeps returning 200s instead of blocks. For the API-driven work in this guide, though, a free token and a sensible request rate cover nearly every use case. Keep the scraper-API approach in reserve for the high-volume HTML jobs the REST API cannot serve.

For the broader fundamentals behind everything here, see the web scraping guide. It covers headers, pagination, parsing, and the etiquette that keeps your scrapers running.