Quick answer: There is no single best PDF parser, because there are two kinds of PDF. Digital-born PDFs have a text layer: parse them with PyMuPDF (fastest) or pdfplumber (best for tables). Scanned PDFs are just pictures of text: they need OCR first, and ocrmypdf plus Tesseract is the standard free route. Fetch the file with proper headers, check whether it has a text layer, then pick the tool. For LLM pipelines, extract to text or markdown and chunk before prompting.
A surprising amount of the web's most valuable content lives in PDFs: government reports, financial filings, research papers, spec sheets, court documents. Scraping PDFs is a different sport from scraping HTML, and the tooling questions come up constantly. Here's the map.
First question: text layer or scan?
Everything downstream depends on this. A digital-born PDF (exported from Word, LaTeX, a browser) contains actual text objects you can extract directly. A scanned PDF contains images of pages, and "extracting text" from it without OCR returns nothing.
Check programmatically before you commit to a parsing strategy:
import fitz # PyMuPDF, install as: pip install pymupdf
def has_text_layer(path: str, sample_pages: int = 5) -> bool:
doc = fitz.open(path)
chars = sum(len(page.get_text()) for page in doc[:sample_pages])
return chars > 100 # near-zero text on sampled pages means it's a scan
print(has_text_layer("report.pdf"))
Mixed files exist too: a digital report with scanned appendices. If completeness matters, check per page, not per file.
Fetching PDFs from the web properly
Before parsing comes fetching, and PDFs have their own gotchas. Some servers block naked clients, some redirect through viewer pages, and some serve HTML error pages with a 200 status that will crash your parser downstream.
import requests
resp = requests.get(
"https://example.com/reports/annual-2025.pdf",
headers={"User-Agent": "Mozilla/5.0 (compatible; MyBot/1.0)"},
timeout=30,
)
resp.raise_for_status()
ctype = resp.headers.get("Content-Type", "")
is_pdf = resp.content[:5] == b"%PDF-"
if not is_pdf:
raise ValueError(f"Not a PDF (Content-Type: {ctype})")
with open("annual-2025.pdf", "wb") as f:
f.write(resp.content)
Checking the magic bytes (%PDF-) matters more than the Content-Type header, which is frequently wrong. For discovering PDFs in the first place (say, every filing linked from an investor-relations page), I fetch the hosting page as markdown with link.sc and pull the .pdf links from it, since links survive the markdown conversion with their URLs intact:
curl https://link.sc/v1/fetch \
-H "Authorization: Bearer lsc_your_api_key" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/investor-relations", "format": "markdown"}'
That also sidesteps the anti-bot layer that many corporate sites put in front of their document libraries.
The tool lineup
For text-layer PDFs in Python, four libraries cover nearly every job:
| Tool | Speed | Strengths | Weaknesses |
|---|---|---|---|
| PyMuPDF (fitz) | Very fast | Speed, layout info, images, huge files | AGPL license (commercial license available) |
| pdfplumber | Slow | Best table extraction, precise positioning | Painful on 500-page documents |
| pypdf | Moderate | Pure Python, no binary deps, merging/splitting | Weakest text extraction quality |
| unstructured | Slow | Partitions into typed elements (title, list, table), LLM-pipeline friendly | Heavy install, many dependencies |
My defaults: PyMuPDF when I want raw text fast, pdfplumber when tables matter, unstructured when I'm feeding a RAG pipeline and want element types preserved. Basic extraction with PyMuPDF:
import fitz
doc = fitz.open("annual-2025.pdf")
text = "\n\n".join(page.get_text() for page in doc)
One honest warning: all of these struggle with multi-column layouts. PDF stores text in draw order, not reading order, so a two-column paper can extract as interleaved gibberish. PyMuPDF's get_text("blocks") plus sorting by position helps, but verify on your actual documents.
Tables need their own approach
Tables are where generic text extraction quietly fails: the cells extract fine as words, but the row and column relationships evaporate. pdfplumber is the best free tool here because it uses the ruling lines and character positions to reconstruct the grid:
import pdfplumber
with pdfplumber.open("annual-2025.pdf") as pdf:
for page in pdf.pages:
for table in page.extract_tables():
for row in table:
print(" | ".join(cell or "" for cell in row))
Emitting tables as markdown pipe rows (like above) is deliberately useful: if the destination is an LLM, markdown tables preserve the row/column structure in a format models handle well.
For borderless tables (whitespace-aligned, no ruling lines), pass table_settings={"vertical_strategy": "text"} and expect to tune. Camelot is an alternative worth trying when pdfplumber's guesses miss.
Scanned PDFs: OCR
If there's no text layer, OCR is mandatory. The standard free stack is ocrmypdf, which wraps Tesseract and writes the recognized text back into the PDF as an invisible layer, so every tool above works afterward:
pip install ocrmypdf
ocrmypdf --language eng --skip-text scanned.pdf searchable.pdf
--skip-text leaves pages that already have text alone, which handles mixed documents gracefully. Expect OCR quality to track scan quality: clean 300 DPI scans come out nearly perfect, phone photos of stapled pages do not. For hard cases (handwriting, degraded scans, complex forms), commercial OCR from the big cloud vendors is measurably better than Tesseract, at a per-page price.
Feeding PDF text to an LLM
Once you have text, three things make it useful to a model rather than merely present in the prompt:
Chunk it. A 200-page PDF exceeds any sensible single prompt. Split on structure (sections, headings, pages) rather than fixed character counts, so chunks are self-contained. I covered strategies in what chunkers actually do.
Keep structure as markdown. Convert headings and tables to markdown instead of flat text. Structure is what lets the model answer "what was Q3 revenue" from a table instead of hallucinating around a word soup. The token math is in token optimization for LLM web data.
Carry metadata. Prefix each chunk with the document title, page number, and section. Citations become possible and retrieval improves.
The pipeline that works: fetch, verify magic bytes, detect text layer, OCR if needed, extract with the right tool, convert tables to markdown, chunk with metadata, then prompt. Every step is boring, and skipping any of them is where PDF projects go to die.
Building a pipeline that mixes webpages and documents? link.sc turns any URL into clean, LLM-ready content with one API call. Start free.