Quick answer: CSS selectors are patterns that match elements in an HTML document. In scraping, you use them to point a parser at exactly the data you want: .price grabs elements with the class price, article h2 grabs every h2 inside an article, and a[href^="/product/"] grabs product links. Master a handful of selector types and combinators and you can target almost anything on a page without writing brittle, position-based code.
CSS selectors are the same patterns that stylesheets use to decide what to style, which means the browser and your scraper speak the same language. Below is a working reference organized from simplest to most powerful.
The basic selectors
These three cover most of what you will target.
| Selector | Matches | Example |
|---|---|---|
tag |
All elements of that type | p matches every paragraph |
.class |
Elements with that class | .price matches class="price" |
#id |
The element with that id | #main matches id="main" |
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "html.parser")
soup.select("p") # every <p>
soup.select(".price") # every element with class "price"
soup.select_one("#main") # the single element with id "main"
select returns a list of every match; select_one returns the first match or None. IDs are meant to be unique on a page, so #id is a good anchor when one exists.
Attribute selectors
Classes and IDs are just special attributes. You can match on any attribute, which is invaluable when a site does not give useful class names.
| Selector | Matches |
|---|---|
[data-id] |
Elements that have a data-id attribute |
[type="email"] |
Elements whose type is exactly email |
[href^="/product/"] |
href that starts with /product/ |
[href$=".pdf"] |
href that ends with .pdf |
[class*="col-"] |
class that contains col- |
# Every link pointing into the product section
soup.select('a[href^="/product/"]')
# Elements carrying a data attribute, regardless of its value
soup.select("[data-testid]")
The ^=, $=, and *= operators (starts-with, ends-with, contains) are the ones you reach for constantly when scraping real sites, because URLs and generated class names follow predictable prefixes and suffixes.
Combinators
Combinators describe relationships between elements. This is how you say "the price inside this specific card" rather than "any price anywhere."
| Combinator | Meaning | Example |
|---|---|---|
A B (space) |
B anywhere inside A |
article .price |
A > B |
B that is a direct child of A |
ul > li |
A + B |
B immediately after A (adjacent sibling) |
h2 + p |
A ~ B |
Any B after A (general sibling) |
h2 ~ p |
# Direct children only: top-level list items, not nested ones
soup.select("ul.menu > li")
# The paragraph right after each heading
soup.select("h2 + p")
The distinction between descendant (space) and child (>) matters when structures nest. ul > li skips list items buried inside a sub-list; ul li grabs them all. Choose based on whether you want the whole subtree or just the top layer.
Pseudo-classes
Pseudo-classes match by position or state. For scraping, the positional ones are the useful set.
| Selector | Matches |
|---|---|
:first-child |
An element that is the first child of its parent |
:last-child |
The last child of its parent |
:nth-child(n) |
The nth child (1-indexed) |
:nth-child(odd) / :nth-child(even) |
Alternating children |
:not(selector) |
Elements that do not match the inner selector |
# The first cell of every table row (often the label column)
soup.select("tr td:first-child")
# Every row except the header row
soup.select("tr:not(.header)")
nth-child is handy for tables and repeated layouts, but it is also one of the more fragile selectors, because it breaks the moment a row is inserted or reordered. Use it when position is genuinely meaningful, not as a shortcut for "the one I saw in position three."
Building selectors that resist layout changes
The difference between a scraper that runs for a year and one that breaks next week is usually selector choice. A few principles:
- Prefer stable attributes over position.
[data-testid="price"]survives redesigns;div > div > span:nth-child(2)does not. Test attributes and semantic IDs change far less often than DOM structure. - Anchor on meaning, not styling. Class names like
.text-red-500describe appearance and get swapped when the design changes. A class like.product-pricedescribes purpose and tends to stick. - Keep selectors as short as the page allows. Every extra step in
body > div > main > section > article > pis another thing that can break. Find the nearest stable ancestor and select from there. - Avoid auto-generated class names. Framework-generated classes like
.css-1a2b3cchange on every build. Match on a prefix ([class^="css-"]) only as a last resort, and prefer a nearbydata-attribute. - Do not trust "Copy selector" verbatim. DevTools generates a full positional path. Use it to find the element, then simplify to the shortest stable selector.
To understand why the element you are selecting might not even be in the source HTML, see what is the DOM for web scraping.
CSS selectors vs XPath
CSS selectors are not the only way to target elements. XPath is the main alternative, and each has strengths.
| CSS selectors | XPath | |
|---|---|---|
| Readability | Concise, familiar from styling | Verbose but very expressive |
| Select by text content | No native support | Yes (//button[text()="Buy"]) |
| Traverse to a parent | No | Yes (.. and ancestor::) |
| Attribute matching | Strong (^=, $=, *=) |
Strong |
| Learning curve | Gentle | Steeper |
The practical rule: use CSS selectors by default because they are shorter and cover most cases. Switch to XPath when you need to match on visible text or walk up the tree to a parent, neither of which CSS can do. For the XPath side of the story, see the XPath cheat sheet for web scraping.
Selectors are only as good as the HTML
A selector can only match content that is actually present in what you parse. If a page renders its data with JavaScript, a plain fetch returns a shell and your carefully chosen selectors match nothing. Rendering the page first, then extracting, is what makes selectors reliable on modern sites. A fetch API can render and return clean content for you:
curl https://link.sc/v1/fetch \
-H "Authorization: Bearer lsc_..." \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/products", "format": "markdown"}'
With the page fully rendered, the content your selectors expect is present, and you can also get it back as clean markdown when you do not need element-level targeting at all. See how link.sc handles rendering in the docs.
The bottom line
CSS selectors give you a compact language for pointing at exactly the elements you want. Learn the basics (tag, class, id), the attribute operators, the combinators, and the positional pseudo-classes, and you can target nearly anything. Build selectors around stable attributes and meaning rather than position and styling, reach for XPath when you need text matching or parent traversal, and make sure the content is actually rendered before you count on any selector matching it.
Want your selectors to have real content to match? Start free with link.sc and fetch fully rendered pages as clean markdown.