← All posts

How to Download All PDFs From a Website (3 Methods That Work)

GUIDES · JUL 19, 2026 · 6 MIN READ

Quick answer: the fastest way to download every PDF from a website is wget -r -l 5 -A pdf -np https://example.com/, which crawls the site and keeps only PDF files. When wget misses files (JavaScript-rendered links, PDFs served without a .pdf extension, bot protection), you need a small file crawler: a script that walks the site's pages, collects document links, and downloads them one by one. Both approaches are below, with working code.

This comes up constantly: a government agency publishes 400 meeting minutes as PDFs, a manufacturer hosts every product datasheet, a university department posts years of course syllabi. You want the whole set, and clicking 400 links is not a plan.

Before You Crawl: Two Shortcuts Worth 30 Seconds

Don't write code until you've checked whether the site already hands you the list.

Shortcut 1: a search engine dork. Search Google for:

site:example.com filetype:pdf

This shows every PDF Google has indexed on that domain. It's often the fastest way to gauge scope (12 files or 12,000?) and it catches documents on subdomains you didn't know existed. The catch: Google only shows what it indexed, and it caps how many results you can page through. Treat it as reconnaissance, not the download mechanism.

Shortcut 2: the sitemap. Try example.com/sitemap.xml. Some sites list document URLs directly in the sitemap, and grabbing URLs from XML is far easier than crawling HTML. If the sitemap has the PDFs, you can skip crawling entirely and go straight to the download loop at the end of this post.

If neither shortcut gives you a complete list, crawl.

Method 1: The wget One-Liner

wget has done recursive, file-type-filtered downloads since the 90s:

wget -r -l 5 -A pdf -np -w 1 -P ./pdfs https://example.com/docs/

What each flag does:

Flag Meaning
-r Recursive: follow links to other pages
-l 5 Crawl at most 5 links deep
-A pdf Accept only files ending in .pdf (HTML is fetched for links, then deleted)
-np No parent: stay under /docs/, don't wander up the site
-w 1 Wait 1 second between requests
-P ./pdfs Save everything into a pdfs folder

For a plain, server-rendered site, this is genuinely all you need. I'd estimate it fully solves the problem for something like half of the sites you'll point it at.

The other half fail for predictable reasons:

  • Links rendered by JavaScript. wget doesn't run JS, so a React-powered document library looks empty to it.
  • PDFs behind extensionless URLs. A link like /download?id=8842 serves a PDF but fails the -A pdf filter, which only looks at the URL.
  • Bot protection. Cloudflare and friends serve wget a challenge page instead of content.
  • No crawl path. If documents are only reachable through a search form, there's no chain of links for wget to follow.

When you hit any of those, move to a script you control.

Method 2: A Python File Crawler

Here's a compact crawler that walks a site, collects PDF links from every page, and downloads them. It handles the extensionless-URL problem by checking the Content-Type header instead of trusting the file extension:

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse
from collections import deque
import os, re

START = "https://example.com/docs/"
DOMAIN = urlparse(START).netloc
OUT = "pdfs"
os.makedirs(OUT, exist_ok=True)

seen_pages, pdf_urls = set(), set()
queue = deque([START])

while queue:
    url = queue.popleft()
    if url in seen_pages:
        continue
    seen_pages.add(url)

    resp = requests.get(url, timeout=20)
    ctype = resp.headers.get("Content-Type", "")

    if "application/pdf" in ctype:
        pdf_urls.add(url)          # the "page" was itself a PDF
        continue
    if "text/html" not in ctype:
        continue

    soup = BeautifulSoup(resp.text, "html.parser")
    for a in soup.select("a[href]"):
        link = urljoin(url, a["href"]).split("#")[0]
        if urlparse(link).netloc != DOMAIN:
            continue               # stay on-site
        if link.lower().endswith(".pdf") or "download" in link.lower():
            pdf_urls.add(link)
        elif link not in seen_pages:
            queue.append(link)

print(f"Found {len(pdf_urls)} candidate PDFs")

for link in sorted(pdf_urls):
    r = requests.get(link, timeout=60)
    if "application/pdf" not in r.headers.get("Content-Type", ""):
        continue                   # candidate wasn't actually a PDF
    name = re.sub(r"[^\w.-]", "_", urlparse(link).path.strip("/")) or "file.pdf"
    if not name.endswith(".pdf"):
        name += ".pdf"
    with open(os.path.join(OUT, name), "wb") as f:
        f.write(r.content)
    print("saved", name)

Two design choices matter here. First, the crawl phase and the download phase are separate loops, the same two-phase pattern I recommend for list crawling: if download 300 of 400 fails, you still have the full URL list and can resume. Second, the final Content-Type check is the source of truth for "is this a PDF," not the URL. I've crawled sites where a third of the documents lived behind /getfile?doc=... URLs that no extension filter would ever catch.

Want Word documents and spreadsheets too? Add their MIME types (application/msword, the openxmlformats types) to the check. The crawler doesn't care what file type it's hunting; that's what makes it a general file crawler rather than a PDF-specific trick.

Add a time.sleep(1) inside both loops for any site you don't own. Hammering a small server with hundreds of rapid requests is how you get IP-banned or start collecting 429 errors.

Method 3: When the Site Fights Back

The Python crawler above shares one blind spot with wget: requests doesn't execute JavaScript and doesn't pass bot checks. Point it at a JS-rendered document portal or a Cloudflare-protected site and you'll get an empty link list or a challenge page.

You have two options. Run a headless browser (Playwright or Selenium) so pages render before you parse them, and accept the setup and babysitting that comes with it. Or swap the fetch call for a managed fetch API and keep the rest of the crawler unchanged. With link.sc, the page-fetching line becomes:

resp = requests.post(
    "https://api.link.sc/v1/fetch",
    headers={"x-api-key": "YOUR_API_KEY"},
    json={"url": url, "render_js": True},
)

The response gives you the rendered HTML after JavaScript has run, so the PDF links that only exist client-side finally show up in your parser. Rendering, retries, and anti-bot handling happen on the API side; your crawl loop and download loop don't change at all. The docs cover the full parameter list.

Note the division of labor: the fetch API is for discovering links on rendered pages. The PDFs themselves are static files, so downloading them with plain requests usually works fine even on sites where the HTML pages are protected.

Is Bulk-Downloading PDFs Legal?

Downloading publicly linked documents for personal use, research, or archiving is generally fine, the same analysis as any web scraping. The places to be careful: documents behind a login (accessing them programmatically can violate terms you agreed to), copyrighted material you plan to republish, and personal data inside the files. And check robots.txt before crawling; staying out of disallowed paths is both polite and the conservative legal position.

One more courtesy that costs nothing: if you'll re-run the crawl on a schedule, cache what you've downloaded and skip files you already have. Most servers send Last-Modified headers that make this trivial, and the site's operator will never have a reason to notice you.

Which Method Should You Use?

Situation Use
Static site, PDFs end in .pdf wget one-liner
Extensionless download URLs Python crawler with Content-Type check
Mixed file types (PDF, DOCX, XLSX) Python crawler with a MIME allowlist
JS-rendered document library Crawler + rendered fetch (headless or API)
Cloudflare or similar protection Crawler + managed fetch API
Just need a quick inventory site:domain filetype:pdf search

Start with the cheapest tool that works and escalate only when it misses files. In practice that means: dork first, wget second, crawler third, rendered fetch last.

Crawling a document library that won't cooperate? Get a free link.sc API key and let the Fetch API render the pages while your script collects the files.