What Is an ETL Pipeline? Extract, Transform, Load Explained

Quick answer: An ETL pipeline is a repeatable process that moves data from one or more sources into a destination where it can be analyzed. ETL stands for Extract (pull raw data from a source), Transform (clean, reshape, and standardize it), and Load (write it into a database, warehouse, or file). Web scraping and API calls are one common way to feed the Extract step, and everything downstream depends on that first pull being clean and reliable.

I have built data pipelines that started as a single cron job and grew into something a whole team depended on. The word "pipeline" sounds heavy, but the idea is simple: data comes in on one end, gets tidied up in the middle, and lands somewhere useful on the other end. ETL is just a name for the three stages of that journey.

This post explains each stage, the difference between ETL and ELT, and where scraping the web fits in.

The Three Stages

An ETL pipeline has three parts, and the acronym is the order they run in.

Extract

Extract is the "get the raw data" step. Your sources might be a production database, a third-party API, a CSV a partner emails you, or a set of public web pages you fetch and parse. The goal here is narrow: pull the data out reliably and completely, without losing rows and without changing anything yet.

Extraction is usually the messiest step because you do not control the source. APIs rate-limit you, CSVs arrive with inconsistent columns, and web pages change their markup without warning. A lot of pipeline failures trace back to a brittle extract.

Transform

Transform is where raw data becomes usable data. Typical work here includes:

Parsing dates into a single format
Trimming whitespace and fixing encodings
Renaming and reordering columns
Deduplicating rows
Joining data from two sources
Filtering out records you do not need
Converting types (a price string like "$19.00" into a number 19.0)

This is the step where business logic lives. Two teams pulling the same raw data can transform it differently depending on what they need.

Load

Load writes the transformed data into its destination: a Postgres table, a data warehouse like BigQuery or Snowflake, or a plain file on disk. Loads come in two flavors. A full load replaces everything each run. An incremental load only writes new or changed records, which is faster and cheaper once your dataset is large.

ETL vs ELT

You will see both ETL and ELT. The letters are the same; the order of the last two changes, and that changes a lot.

	ETL	ELT
Order	Extract, Transform, Load	Extract, Load, Transform
Where transform happens	In a separate processing step before loading	Inside the destination warehouse, after loading
Best when	Destination is expensive to compute in, or data must be cleaned before it lands	Warehouse is powerful and cheap to query (BigQuery, Snowflake)
Raw data kept?	Often discarded after transform	Raw data usually preserved in the warehouse
Typical era	Classic on-premise data warehousing	Modern cloud warehouses

The short version: ELT became popular because cloud warehouses got fast and cheap enough to do heavy transformation right where the data lands. If you are starting fresh with a modern warehouse, ELT is often the simpler choice. If your destination is not built for heavy compute, ETL still makes sense.

Where Web Scraping Fits

Not all data lives in a tidy API. Sometimes the numbers you need are sitting on public web pages: competitor pricing, product catalogs, job listings, published research, government records. In that case, scraping the web is your Extract step.

This is where a lot of pipelines get fragile. Fetching pages reliably means handling redirects, JavaScript rendering, rate limits, and markup that shifts under you. If you want to see the range of tools people use for that, we compared them in Python web scraping libraries in 2026.

A fetch-and-search API can collapse the messy part of extraction into one call. Instead of running a headless browser, rotating proxies, and writing parsers, you send a URL and get clean content back. Here is what the Extract step looks like against link.sc:

import requests

LINKSC_KEY = "lsc_your_key_here"

def extract(url: str) -> str:
    resp = requests.post(
        "https://api.link.sc/v1/fetch",
        headers={"x-api-key": LINKSC_KEY},
        json={"url": url, "format": "markdown"},
        timeout=60,
    )
    resp.raise_for_status()
    return resp.json()["content"]

raw = extract("https://example.com/products")
print(raw[:500])

Note the auth header is x-api-key, not Authorization: Bearer. The response comes back as { "content": "..." } in clean markdown, which is far easier to transform than raw HTML.

A Simple End-to-End Pipeline

Here is a tiny but complete ETL pipeline. It extracts a page, transforms the text into structured records, and loads them into a SQLite database. The shape is the same whether you scale it to millions of rows or run it once on a laptop.

import re
import sqlite3
import requests

LINKSC_KEY = "lsc_your_key_here"

# 1. EXTRACT
def extract(url: str) -> str:
    resp = requests.post(
        "https://api.link.sc/v1/fetch",
        headers={"x-api-key": LINKSC_KEY},
        json={"url": url, "format": "markdown"},
        timeout=60,
    )
    resp.raise_for_status()
    return resp.json()["content"]

# 2. TRANSFORM
def transform(markdown: str) -> list[dict]:
    records = []
    # find lines like "- Widget A: $19.00"
    for line in markdown.splitlines():
        match = re.match(r"-\s*(.+?):\s*\$([\d.]+)", line)
        if match:
            name = match.group(1).strip()
            price = float(match.group(2))  # "$19.00" -> 19.0
            records.append({"name": name, "price": price})
    return records

# 3. LOAD
def load(records: list[dict], db_path: str = "products.db") -> None:
    conn = sqlite3.connect(db_path)
    conn.execute(
        "CREATE TABLE IF NOT EXISTS products (name TEXT PRIMARY KEY, price REAL)"
    )
    conn.executemany(
        "INSERT OR REPLACE INTO products (name, price) VALUES (:name, :price)",
        records,
    )
    conn.commit()
    conn.close()

if __name__ == "__main__":
    raw = extract("https://example.com/products")
    clean = transform(raw)
    load(clean)
    print(f"Loaded {len(clean)} rows")

Each function maps to one letter of ETL. That separation is the whole point: when the extract breaks, you fix extract and leave the rest alone. For more on the Load step and choosing between CSV, JSON, and a database, see how to export scraped data to CSV, JSON, or a database.

Tools, Conceptually

You do not have to hand-roll everything. The ecosystem breaks down roughly like this:

Job	Examples of tools
Orchestration (scheduling, retries, dependencies)	Airflow, Dagster, Prefect
Managed connectors (SaaS sources to warehouse)	Fivetran, Airbyte
Transformation	dbt, plain SQL, pandas
Warehouse / destination	BigQuery, Snowflake, Postgres, DuckDB
Web extraction	requests plus a parser, headless browsers, or a fetch API

For a first pipeline, do not reach for orchestration frameworks on day one. A single script and a cron job is a real pipeline. Add Airflow or Dagster when you have several pipelines with dependencies between them and you need retries and visibility.

A Note on Sourcing Data Responsibly

When your Extract step pulls from the web, stay on the right side of the line. Scrape public data only, honor robots.txt and any published rate limits, do not log in to bypass access controls, and space out your requests so you do not degrade someone else's site. A pipeline that hammers a source is both rude and easy to block. Treat other people's servers the way you would want yours treated: a few well-behaved requests, cached where possible, is almost always enough.

The Bottom Line

An ETL pipeline is just Extract, Transform, Load: pull raw data, clean it, and put it somewhere useful. ELT swaps the order to transform inside a modern warehouse. When your data lives on the open web, extraction is scraping, and the reliability of that first step decides whether the whole pipeline holds up. Keep the three stages as separate functions, start small, and only add heavy tooling when the scale demands it.

Want the Extract step to be one clean API call instead of a browser farm? link.sc turns any URL into structured markdown or JSON. Start free with 500 credits a month.