← All posts

Web Scraping Example: A Step-by-Step Tutorial with Real Code

ENGINEERING · JUL 14, 2026 · 4 MIN READ

Quick answer: Here is a complete web scraping example in Python: it fetches a page, extracts every book's title and price, and saves a CSV. Install the dependencies (pip install requests beautifulsoup4), paste, and run:

import csv
import requests
from bs4 import BeautifulSoup

resp = requests.get("https://books.toscrape.com/")
soup = BeautifulSoup(resp.text, "html.parser")

rows = []
for book in soup.select("article.product_pod"):
    rows.append({
        "title": book.h3.a["title"],
        "price": book.select_one(".price_color").text.strip(),
    })

with open("books.csv", "w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=["title", "price"])
    writer.writeheader()
    writer.writerows(rows)

print(f"Saved {len(rows)} books to books.csv")

Twenty lines, three steps: fetch, parse, export. The rest of this tutorial explains each step so you can adapt it to any site, shows the same example in JavaScript, and covers what to do when a real site fights back.

Step 1: Fetch the HTML

requests.get(url) downloads the page's HTML, exactly what you'd see with View Source in your browser. We're using books.toscrape.com, a site built for scraping practice, so nobody minds.

One habit to build from day one: check the response before parsing.

resp = requests.get(url, timeout=10)
resp.raise_for_status()  # throws on 4xx/5xx instead of parsing an error page

Silent failures (happily parsing a 404 page or a "please slow down" message) are the classic beginner bug.

Step 2: Parse and Find Your Data with Selectors

BeautifulSoup turns HTML text into a searchable tree. The skill is writing CSS selectors, and you get them from your browser: right-click the data → Inspect → look at the surrounding structure.

On our practice site, each book looks like:

<article class="product_pod">
  <h3><a title="A Light in the Attic" href="...">A Light in the ...</a></h3>
  <p class="price_color">£51.77</p>
</article>

Which maps directly to the selectors in the script:

  • soup.select("article.product_pod") → every book card
  • book.h3.a["title"] → the title attribute
  • book.select_one(".price_color").text → the price text

That's genuinely all parsing is: find the repeating container, then pick fields out of each one. When a scrape "breaks" months later, it's almost always because the site changed its HTML and these selectors need updating.

Step 3: Export It Somewhere Usable

We wrote CSV because spreadsheets are the universal customer. json.dump(rows, f) works just as well, and inserting into a database is the same loop. The point is that scraped data should land somewhere structured immediately, not in print statements.

The Same Example in JavaScript

If you live in Node.js (npm install cheerio):

import * as cheerio from "cheerio";

const resp = await fetch("https://books.toscrape.com/");
const $ = cheerio.load(await resp.text());

const rows = $("article.product_pod")
  .map((_, el) => ({
    title: $(el).find("h3 a").attr("title"),
    price: $(el).find(".price_color").text().trim(),
  }))
  .get();

console.log(rows);

Same three steps, same selectors. Language choice barely matters at this scale. We have a fuller guide to crawling with Node.js if you want to go multi-page.

Where Real Sites Differ from the Tutorial

Run this pattern against a modern commercial site and you'll hit one of three walls, usually within the hour:

Empty results. Your selectors are right, but the data isn't in the HTML: the site renders it with JavaScript after load. Confirm by checking View Source (not Inspect): if the data's not there, plain requests can never see it. You need JS rendering.

A challenge page instead of content. Cloudflare or similar decided you're a bot. Plain HTTP clients fail these checks by default.

429 errors. You're being rate-limited for going too fast. It's fixable with delays and backoff, covered in our 429 guide.

The first two are why "the tutorial worked, but the real site doesn't" is the universal scraping experience. The pragmatic fix is to keep your parse-and-export code and replace the fetch step with a service that handles rendering and anti-bot measures:

resp = requests.post(
    "https://api.link.sc/v1/fetch",
    headers={"x-api-key": "YOUR_API_KEY"},
    json={"url": "https://example.com/products", "render_js": True, "format": "markdown"},
)
content = resp.json()["content"]

Everything you learned above still applies; you've just outsourced the hostile half of the problem. You can try URLs interactively in the link.sc playground before writing any code.

Rules for Scraping Politely

Even on easy sites: add a delay between requests (a second is friendly), set a User-Agent that identifies you, check robots.txt, and don't scrape personal data. Public product listings are one thing; people's profiles are legally and ethically different terrain. See is web scraping legal? for where the lines sit.

Your Next Steps

  1. Run the example above verbatim.
  2. Point it at a real site you care about and rewrite the two selectors.
  3. Add pagination with a loop over ?page=N; that's list crawling.
  4. When you meet JavaScript rendering or blocks, swap the fetch layer instead of fighting it.

That sequence takes most people from zero to a useful dataset in an afternoon.

Skip the walls entirely: get a free link.sc API key and fetch any page (rendered, unblocked, as clean Markdown) from day one.