Quick answer: The DOM (Document Object Model) is the live, in-memory tree that a browser builds from a page's HTML. Every element, attribute, and piece of text becomes a node in that tree, and scripts on the page can add, remove, or change nodes after the page loads. For scraping, the key fact is this: the HTML you download with a simple request is the source, but the DOM is what the browser actually shows, and on modern sites those two can be very different.
Understanding the gap between source HTML and the rendered DOM is the single most useful thing you can know when a scraper "sees" nothing while your browser clearly shows the data.
The DOM is a tree
When a browser loads a page, it parses the HTML into a tree of nodes. A document like this:
<article>
<h1>Widget Pro</h1>
<p class="price">$49</p>
</article>
becomes a tree the browser holds in memory:
article
├── h1
│ └── "Widget Pro"
└── p.price
└── "$49"
Each element is a node. Nodes have parents, children, and siblings. Attributes like class="price" hang off their element. This tree is what CSS styles, what JavaScript manipulates, and what selectors traverse. The HTML text file is just the recipe; the DOM is the cake.
Source HTML vs the rendered DOM
Here is where scrapers get tripped up. When you fetch a URL with a plain HTTP client, you get the raw HTML exactly as the server sent it, before any JavaScript runs.
import requests
html = requests.get("https://example.com/product/42").text
# `html` is the source. If the price is injected by JavaScript,
# it is NOT in this string.
Many modern sites send a nearly empty HTML shell and then use JavaScript to fetch data and build the visible content. The price, the reviews, the product list, all of it gets inserted into the DOM after load. Your browser runs that JavaScript, so you see a full page. Your requests call does not run JavaScript, so it sees the empty shell.
| Source HTML | Rendered DOM | |
|---|---|---|
| What it is | The raw text the server sent | The live tree after scripts run |
| Contains JS-injected content | No | Yes |
What requests / curl gets |
This | Not this |
| What a browser shows | Starting point only | This |
| What "View Source" shows | This | Not this (use DevTools instead) |
This is why "View Source" in your browser can look completely different from what is on the screen. View Source shows the original HTML; the Elements panel in DevTools shows the current DOM.
Why the difference matters
If you write a scraper that parses source HTML and the data you want is JavaScript-rendered, your selectors will match nothing. The data is not missing because you wrote the selector wrong; it is missing because it was never in the source, only in the DOM.
You have two ways to close the gap:
- Find the underlying data request. Often the JavaScript is fetching JSON from an API endpoint. If you can locate that request in the Network tab, you can call it directly and skip HTML parsing entirely.
- Render the page. Run the page in a real (usually headless) browser so the JavaScript executes and builds the full DOM, then read from that. This is slower and heavier but works when there is no clean API to hit.
For the full treatment of option two, see how to scrape JavaScript-rendered websites.
Inspecting the DOM with DevTools
Your browser's developer tools are the fastest way to understand what you are scraping. Right-click any element on a page and choose "Inspect" to open the Elements panel, which shows the current DOM tree, not the source.
A few habits pay off:
- Elements panel shows the live DOM. Expand nodes to see the tree structure and find the element holding your data.
- Right-click a node then "Copy" then "Copy selector" to get a CSS selector that targets it. Treat this as a starting point and simplify it; generated selectors are often brittle.
- Network tab reveals the requests the page makes after load. Filter to "Fetch/XHR" to spot JSON API calls that might give you the data directly.
- Compare View Source with the Elements panel. If the data is in Elements but not in View Source, it is JavaScript-rendered, and a plain fetch will not see it.
Traversing the DOM in code
Once you have the DOM (either from source HTML that already contains your data, or from a rendered page), a parser lets you walk the tree with selectors.
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "html.parser")
# Traverse by selector
title = soup.select_one("article h1").get_text(strip=True)
price = soup.select_one("article p.price").get_text(strip=True)
print(title, price) # "Widget Pro $49"
select_one walks the parsed tree looking for a node matching the CSS selector, exactly the way the browser matches CSS rules against the DOM. The parser gives you the same tree structure the browser has, so the same selectors work.
Getting the rendered DOM without running a browser yourself
Running a headless browser to get the rendered DOM is effective but operationally heavy: you manage browser binaries, memory, timeouts, and the anti-bot friction that comes with automated browsers. A fetch API can do the rendering for you and hand back clean content built from the fully rendered DOM.
curl https://link.sc/v1/fetch \
-H "Authorization: Bearer lsc_..." \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/product/42", "format": "markdown"}'
Because the page is rendered before extraction, JavaScript-injected content is present in the result, and you get it back as markdown instead of a DOM tree you have to traverse yourself. See how link.sc handles rendering in the docs.
The bottom line
The DOM is the browser's live tree built from HTML and modified by JavaScript. Source HTML is only the starting point; on JavaScript-heavy sites the data you want lives in the DOM, not the source, which is why plain fetches come back empty. Use DevTools to tell the two apart, look for an underlying data API first, and render the page when you must. Whether you render it yourself or use a fetch API, the goal is the same: get to the DOM the browser actually shows.
Tired of empty scrapes on JavaScript-heavy pages? Try link.sc free and get the fully rendered content as clean markdown.