← All posts

How to Scrape Wikipedia Data (The Right Way)

GUIDES · JUL 18, 2026 · 5 MIN READ

Quick answer: Do not scrape Wikipedia by hammering article pages. Use the official MediaWiki Action API for live queries or download the free database dumps for bulk work. Both are fast, sanctioned, and free. Parsing rendered article HTML only makes sense for the small slice of cases the API and dumps do not cover, and even then you must credit authors under the CC BY-SA license.

Wikipedia is one of the most useful data sources on the internet, and it is one of the few large sites that actively wants you to have its data. That is why "scraping" it the crude way (looping over wikipedia.org/wiki/... pages with a browser emulator) is almost always the wrong call. There is a cleaner path.

Start With the MediaWiki API

Every Wikipedia language edition exposes the MediaWiki Action API at https://en.wikipedia.org/w/api.php. It gives you article content, metadata, revision history, categories, links, and search, all as structured JSON. No HTML parsing, no guessing at page layout.

Here is a plain fetch of an article extract in Python:

import requests

url = "https://en.wikipedia.org/w/api.php"
params = {
    "action": "query",
    "format": "json",
    "prop": "extracts",
    "titles": "Web scraping",
    "explaintext": 1,          # plain text, no HTML
    "redirects": 1,            # follow "Web Scraping" -> "Web scraping"
}
headers = {
    # MediaWiki asks that you identify your client with contact info
    "User-Agent": "MyResearchBot/1.0 ([email protected])"
}

resp = requests.get(url, params=params, headers=headers, timeout=30)
pages = resp.json()["query"]["pages"]
for page_id, page in pages.items():
    print(page["title"])
    print(page["extract"][:500])

A few things worth knowing about the API:

  • Set a real User-Agent. The Wikimedia User-Agent policy expects a descriptive string with a way to contact you. Generic or empty agents can get rate limited or blocked.
  • Batch your titles. You can request up to 50 titles in one call ("titles": "A|B|C"), which cuts your request count dramatically.
  • Use continue tokens for large result sets instead of scraping page after page.

There is also a newer REST API at /api/rest_v1/ and the Wikimedia Enterprise API for high-volume commercial use, but the Action API covers the vast majority of needs.

Use Database Dumps for Bulk Work

If you want all of Wikipedia (or a whole category tree, or every article in a language), do not crawl it. Wikimedia publishes full database dumps at dumps.wikimedia.org, refreshed roughly twice a month.

The file most people want is the current article text, named something like enwiki-latest-pages-articles.xml.bz2. It is a compressed XML stream of every article, and you parse it locally with zero requests to Wikipedia's servers.

import bz2
import xml.etree.ElementTree as ET

# Stream the dump so you never load 20+ GB into memory at once
path = "enwiki-latest-pages-articles.xml.bz2"
ns = "{http://www.mediawiki.org/xml/export-0.11/}"

with bz2.open(path, "rb") as fh:
    for event, elem in ET.iterparse(fh, events=("end",)):
        if elem.tag == f"{ns}page":
            title = elem.findtext(f"{ns}title")
            text = elem.findtext(f"{ns}revision/{ns}text")
            if title and text:
                handle(title, text)   # your logic here
            elem.clear()              # free memory as you go

The dump content is wikitext (Wikipedia's markup), not clean prose. You will want a parser like mwparserfromhell to strip templates and markup, or wikiextractor to bulk-convert dumps to plain text.

When Parsing Article HTML Is Fine

The API and dumps cover almost everything. But there are legitimate cases where you want the rendered page:

  • You need the article exactly as a reader sees it, with tables and infoboxes already laid out.
  • You are pulling a handful of pages and do not want to learn wikitext.
  • You need something the API does not model cleanly, like a specific rendered table.

For that, fetching the rendered HTML and converting it to clean markdown is reasonable, as long as you stay polite (low request rate, cache aggressively, honor robots.txt). This is exactly what a fetch API is for. With link.sc, one call returns parsed markdown so you skip the HTML wrangling:

curl -X POST https://api.link.sc/v1/fetch \
  -H "x-api-key: lsc_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://en.wikipedia.org/wiki/Web_scraping",
    "format": "markdown"
  }'

The response is { "content": "..." } with the article as markdown, ready to feed into an LLM or a document store. If you specifically need tables or infobox fields as structured data, see our guide on how to extract structured JSON from any webpage.

API vs Dumps vs HTML: Which to Use

Need Best tool Why
A few live articles, structured MediaWiki Action API JSON, no parsing, always current
Search across Wikipedia Action API (list=search) Native search, ranked results
Revision history / metadata Action API Purpose-built endpoints
Every article, offline analysis Database dumps Zero live load, reproducible
Rendered page as clean markdown Fetch API Skips wikitext, ready for LLMs

The rule of thumb: live and small, use the API; bulk and offline, use the dumps; rendered content for an LLM, fetch it.

Licensing and Attribution (Do Not Skip This)

Wikipedia text is licensed under Creative Commons Attribution-ShareAlike (CC BY-SA), and some media has other licenses. That has real consequences if you republish:

  • Attribution. You must credit Wikipedia and, where practical, link back to the source article and its contributors. A link to the article page satisfies the contributor-credit requirement.
  • ShareAlike. If you remix or adapt the text and redistribute it, your derivative has to carry a compatible CC BY-SA license.
  • Media is separate. Images and other files each have their own license (some CC, some public domain, some fair use). Check per file; do not assume the article license covers them.

Using the data internally (analysis, training signals, search indexing) is generally fine. Republishing article text on your own site without credit is not. When in doubt, keep the link back to the source.

A Quick Note on Ethics and Rate Limits

Wikipedia runs on donations. Treating it like a free CDN by blasting thousands of requests a second is both rude and self-defeating (you will get rate limited). The whole reason dumps exist is so you never have to do that.

  • Prefer dumps for anything bulk.
  • Batch API calls and cache results.
  • Identify your client honestly with a real User-Agent.
  • Respect robots.txt and any rate signals you get back.

These are the same principles that apply anywhere. If you want the broader picture, our post on ethical web scraping and compliance best practices goes deeper, and the fundamentals of whether web scraping is legal are worth a read before any project.

Putting It Together

For most projects the answer is boring in the best way: the MediaWiki API and database dumps already give you everything, for free, with the site's blessing. Reach for HTML parsing only when you specifically need the rendered page, and let a fetch API handle the messy parts. Credit your sources, keep your request rate sane, and you will have clean Wikipedia data without fighting the site or its license.

Need clean, structured content from Wikipedia or anywhere else on the web? link.sc turns any URL into LLM-ready markdown in one call. Start free with 500 credits a month.