Quick answer: Data extraction is the process of pulling data out of its original source (a website, PDF, database, email, image, or API) and converting it into a structured format you can actually work with, like JSON, CSV, or database rows. It's the "E" in ETL (extract, transform, load), and it's always the first step: you can't analyze, store, or feed data to an AI model until you've extracted it from wherever it lives.
The concept is simple. The practice varies enormously depending on where the data lives, so let's break it down by source, because the source determines the method.
The Five Sources (and How You Extract from Each)
| Source | Method | Typical tools |
|---|---|---|
| Websites | Web scraping / fetch APIs | BeautifulSoup, Scrapy, link.sc |
| APIs | Direct requests | fetch, requests, SDKs |
| Databases | Queries and exports | SQL, CDC tools |
| Documents (PDF, Word, email) | Parsing and OCR | pdfplumber, Tesseract, LLMs |
| Images and scans | OCR / vision models | Tesseract, cloud vision APIs |
APIs are the happy path. If the data owner offers an API, use it: the data arrives already structured, with a contract. Extraction is one authenticated request.
Databases you control are nearly as easy: a SQL query is an extraction. The interesting problems are incremental extraction (only what changed since last run) and not hammering a production database, which is why change-data-capture tools exist.
Websites are the biggest category in practice, because most of the world's information is published as web pages, not APIs. Extracting it means web scraping: fetch the HTML, parse it, pull the fields. This is its own discipline with its own tooling; our scraper tool guide covers the landscape.
Documents and images used to be the hardest category: brittle regex over PDF text, OCR cleanup. LLMs changed this one genuinely: "extract the invoice number, date, and total from this document as JSON" now works remarkably well as a prompt, even on messy layouts.
A Concrete Example: Web Data, End to End
Say you want every job posting from a careers page in a spreadsheet. That's an extraction problem where the source is a website:
import requests
resp = requests.post(
"https://api.link.sc/v1/fetch",
headers={"x-api-key": "YOUR_API_KEY"},
json={"url": "https://example.com/careers", "format": "markdown"},
)
content = resp.json()["content"] # clean Markdown, boilerplate stripped
Notice what happened: unstructured HTML went in, and a clean, readable document came out. From there you either parse the fields with code, or (the increasingly standard move) hand the Markdown to an LLM with a schema and let it structure the records. That two-step (fetch clean content, then structure it) has quietly become the default modern extraction pipeline, and it's exactly the workflow link.sc is built around: the fetch API strips pages down to content so the LLM step is cheap and accurate.
Structured, Semi-Structured, Unstructured
Worth 60 seconds, because it predicts how hard your extraction will be:
- Structured data has a fixed schema: database tables, CSV files. Extraction is trivial; querying is extracting.
- Semi-structured data has organization but no rigid schema: HTML, JSON, XML, email headers. Extraction means navigating the structure: selectors, keys, paths.
- Unstructured data has no machine-readable organization: prose, scanned contracts, photos. Extraction means OCR, NLP, or an LLM.
Most "data extraction" work in the wild is semi-structured: the information visibly has structure (a product card, a table row) but nobody handed you the schema.
What Trips People Up
Volume changes the problem. Extracting one page is a script; extracting 100,000 pages nightly is infrastructure: queues, retries, rate-limit handling, monitoring. Plan for the second shape if the job repeats.
Sources change without notice. Websites redesign, PDFs get new layouts, APIs version. Durable extraction pipelines detect failures loudly (schema validation on the output is the cheapest alarm) rather than silently writing garbage.
Permission matters. Extracting from your own database is your business. Extracting from someone's website has rules: robots.txt, terms of service, and personal-data law. Our guide to whether web scraping is legal covers where the lines are.
How to Actually Start
Match the method to the source and start embarrassingly small:
- Identify where the data lives (usually: a website).
- Check for an API first; five minutes of looking can save weeks.
- Extract one record end to end, by the crudest means available.
- Only then automate: loop, schedule, validate, alert.
Extraction projects die from skipping step 3: building a pipeline before proving the data is gettable and useful.
Most extraction projects start with web data. Create a free link.sc account and turn any URL into clean, structured content with one API call.