Quick answer: BeautifulSoup is a Python library for parsing HTML and pulling data out of it. Install it with pip install beautifulsoup4, fetch a page's HTML (usually with requests), parse it into a BeautifulSoup object, then find the elements you want with find, find_all, or CSS selectors via select. It is fast, forgiving of messy markup, and perfect for static pages, but it does not run JavaScript, so pages that render client-side will come back empty.
BeautifulSoup was the first scraping tool I ever used, and it is still the one I reach for when a page is plain HTML. It does not fetch pages and it does not run scripts. It does exactly one thing well: turn a blob of HTML into a tree you can search. This tutorial takes you from a clean install to a working scraper and is honest about where the library stops.
Install BeautifulSoup
The package is named beautifulsoup4, and you almost always want requests alongside it to actually download pages.
pip install beautifulsoup4 requests
I also recommend installing the lxml parser. It is faster and more robust than Python's built-in parser.
pip install lxml
Parse Your First Page
The pattern is always: fetch, then parse.
import requests
from bs4 import BeautifulSoup
html = requests.get("https://example.com").text
soup = BeautifulSoup(html, "lxml")
print(soup.title.text) # the <title> text
print(soup.h1.text) # the first <h1>
BeautifulSoup(html, "lxml") builds the tree. If you skipped the lxml install, use "html.parser" instead, which ships with Python.
Here are the parser choices side by side:
| Parser | Speed | Install | Notes |
|---|---|---|---|
html.parser |
Medium | Built in | No dependency; fine for small jobs |
lxml |
Fast | pip install lxml |
My default; handles broken HTML well |
html5lib |
Slow | pip install html5lib |
Most lenient; parses like a browser |
find and find_all
These two methods are the core of the library. find returns the first match; find_all returns a list of every match.
# First <a> tag
first_link = soup.find("a")
# Every <a> tag
all_links = soup.find_all("a")
# Filter by attributes
soup.find("div", class_="product") # note: class_ with underscore
soup.find_all("a", href=True) # only <a> tags that have href
soup.find("input", {"name": "email"}) # attribute dict for any attribute
Once you have an element, you read its text and attributes:
link = soup.find("a", class_="detail")
print(link.text) # the visible text
print(link.get_text(strip=True)) # text with whitespace trimmed
print(link["href"]) # attribute value
print(link.get("href")) # same, but returns None if missing
I use link.get("href") over link["href"] in loops because a missing attribute returns None instead of raising a KeyError that kills the whole run.
CSS Selectors With select
If you already know CSS selectors, select and select_one are often cleaner than nested find calls.
soup.select_one("div.product h2") # first match, descendant selector
soup.select("ul.items > li") # direct children
soup.select("a[href^='https']") # attribute starts-with
soup.select("table tr td:nth-of-type(2)") # nth column of every row
select returns a list, select_one returns the first match or None. For anything beyond a single tag lookup, I default to select because the selector reads the way I already think about the DOM.
Navigate the Tree
Sometimes the element you want has no useful class, but its neighbor does. BeautifulSoup lets you walk the tree in every direction.
label = soup.find("span", string="Price")
value = label.find_next_sibling("span") # the span right after it
row = soup.find("td", string="Total")
price = row.parent.find_all("td")[-1] # last cell in the same row
first_item = soup.select_one("ul.items li")
next_item = first_item.find_next("li")
parent, find_next_sibling, find_previous_sibling, find_next, and .children cover almost every "the data is near this landmark" situation.
A Complete End-to-End Example
This scrapes a listing page, pulls a title, price, and link from each card, and skips any card that is missing a field instead of crashing.
import requests
from bs4 import BeautifulSoup
import csv
def scrape(url):
resp = requests.get(url, headers={
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
}, timeout=20)
resp.raise_for_status()
soup = BeautifulSoup(resp.text, "lxml")
rows = []
for card in soup.select("div.product-card"):
title = card.select_one("h2")
price = card.select_one(".price")
link = card.select_one("a")
if not (title and price and link):
continue # incomplete card, skip it
rows.append({
"title": title.get_text(strip=True),
"price": price.get_text(strip=True),
"url": link.get("href"),
})
return rows
if __name__ == "__main__":
data = scrape("https://example.com/products")
with open("products.csv", "w", newline="") as f:
writer = csv.DictWriter(f, fieldnames=["title", "price", "url"])
writer.writeheader()
writer.writerows(data)
print(f"wrote {len(data)} rows to products.csv")
Setting a real User-Agent matters more than people expect. Many sites return a stripped page or a block to the default python-requests agent.
The Big Limitation: No JavaScript
Here is the honest boundary. BeautifulSoup only sees the HTML that requests downloaded. If a page renders its content with JavaScript after load (most modern React, Vue, and Angular apps), that HTML is a nearly empty shell and your find_all returns nothing.
The tell is simple: if print(soup.select("div.product-card")) is an empty list but the products clearly exist in your browser, the content is client-rendered.
You have three ways out:
- Find the underlying API. Open DevTools, watch the Network tab, and often the data arrives as clean JSON you can request directly. No parsing needed.
- Use a real browser. Tools like Playwright render the page first, then you can hand the rendered HTML to BeautifulSoup. See my Playwright web scraping guide.
- Use a fetch API that renders for you. link.sc runs the browser and proxies server-side and returns clean markdown or JSON:
curl https://link.sc/v1/fetch \
-H "Authorization: Bearer lsc_..." \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/products", "format": "markdown"}'
That is the whole trade: BeautifulSoup is a parser, not a renderer. Pair it with something that produces rendered HTML and it stays useful. The parameters are in the link.sc docs.
A Note on Ethics
Stick to public pages, check robots.txt, add a timeout and a delay between requests, and do not hammer a server. If a site offers an API or a data export, use it. It is more stable than scraping and it is what the operator wants you to do.
Where BeautifulSoup Fits
For static, server-rendered pages, BeautifulSoup is hard to beat: small, readable, and forgiving. For a broader survey of the ecosystem, see the Python web scraping libraries for 2026. Learn its selectors well, respect its one real limitation, and it will handle a large share of everyday scraping.
Hitting empty pages because the content is JavaScript-rendered? link.sc returns fully rendered, clean content from any URL in a single call. Get 500 free credits a month.