← All posts

How to Wait for Dynamic Content to Load Before Scraping

GUIDES · JUL 19, 2026 · 6 MIN READ

Quick answer: wait for a specific selector that only appears once the real data has rendered. In Playwright that is page.wait_for_selector(".product-card"), in Puppeteer it is page.waitForSelector(".product-card"). It is the most reliable strategy because it asserts on the exact thing you came for. Network idle is a reasonable fallback when you cannot name a selector, and a fixed sleep() is the option of last resort. Here is why, and how each one fails.

The Problem: "Page Loaded" Does Not Mean "Content Loaded"

On a JavaScript-heavy site, the browser fires its load event long before the content you want exists. The sequence usually looks like this:

  1. The server returns a skeleton HTML document and a JavaScript bundle.
  2. The DOMContentLoaded and load events fire. The page is "loaded" in the browser's eyes.
  3. The JavaScript boots, calls one or more APIs, and waits for JSON to come back.
  4. The framework renders the data into the DOM, sometimes in several passes.

If your scraper grabs the HTML at step 2, you get spinners, skeleton placeholders, or an empty <div id="root">. That is why page.goto(url) followed immediately by page.content() so often returns nothing useful, even though the navigation "succeeded." Waiting is not an optimization. On dynamic pages it is the difference between data and an empty shell. If you are not sure whether your target renders client-side at all, start with how to scrape a JavaScript-rendered website.

So the real question is: what signal tells you the page is actually done? You have three main choices.

Strategy 1: Wait for a Specific Selector (Best Default)

Pick an element that can only exist once the data has arrived, and wait for it:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com/products")
    page.wait_for_selector(".product-card", timeout=15000)
    html = page.content()
    browser.close()

The equivalent in Puppeteer:

await page.goto("https://example.com/products");
await page.waitForSelector(".product-card", { timeout: 15000 });
const html = await page.content();

Why this wins: you are waiting on your success condition, not a proxy for it. When the selector appears, the data you want is in the DOM by definition. It is also fast, because it resolves the instant the element renders instead of padding every request with a worst-case delay.

Two details matter in practice:

  • Choose a data-bearing selector, not a layout one. .container often exists in the empty shell. .product-card, [data-testid="price"], or a row inside the results table only exist after hydration.
  • Beware skeleton screens. Many sites render placeholder cards with the same class as real cards while loading. If that happens, wait for something the placeholder lacks, like non-empty text: page.wait_for_function("document.querySelector('.price')?.textContent.trim().length > 0").

The weakness of selector waits is maintenance. Selectors are site-specific, and when the site ships a redesign your wait breaks along with your extraction logic. For one site that is fine. For a hundred sites it becomes a standing chore.

Strategy 2: Wait for Network Idle (Good Generic Fallback)

Network idle waits until the page has had no in-flight requests (or very few) for a quiet window, typically 500ms:

page.goto("https://example.com/products", wait_until="networkidle")

This is attractive because it needs zero knowledge of the page. That makes it the default choice for generic crawlers that visit arbitrary URLs and cannot know which selector matters on each one.

The failure modes are real, though:

  • It can wait forever. Pages with analytics beacons, chat widgets, long-polling, or open websockets never go quiet. Playwright's own docs discourage networkidle for exactly this reason. Always pair it with a timeout and treat the timeout as a soft signal, not an error.
  • It can fire too early. If the site fetches data in stages (first the list, then prices, then stock), the network can look idle between stages and you capture a half-rendered page.
  • It is slower than a selector wait. You always pay the quiet-window penalty even when the content rendered instantly.

A related, more surgical option is waiting for the specific API response the page depends on: page.wait_for_response(lambda r: "/api/products" in r.url). That combines the precision of a selector wait with network-level knowledge, but it requires you to inspect the site's traffic first, and it breaks when the endpoint changes.

Strategy 3: Fixed Delays (Last Resort)

The tempting one-liner:

page.goto("https://example.com/products")
page.wait_for_timeout(5000)  # hope 5 seconds is enough

A fixed sleep is simple and works on the happy path, which is exactly what makes it dangerous. It fails in both directions at once:

  • Too short sometimes. On a slow network, a cold CDN cache, or a throttled API, 5 seconds is not enough and you silently scrape an unfinished page. No exception, just quietly wrong data.
  • Too long always. On a fast load, the content was ready in 800ms and you burned 4.2 seconds for nothing. Multiply that across ten thousand pages and you have wasted hours of compute and browser time.

Fixed delays have a place: as a small buffer after an interaction (for example, 250ms after a scroll to let a lazy-loaded image settle), or as a stopgap while you find the right selector. As the primary wait strategy for a production scraper, they are the most common source of flaky, silently-incomplete data.

A Practical Decision Rule

  • You know the page and can name a data-bearing element: selector wait, with a timeout of 10 to 20 seconds.
  • You are crawling arbitrary URLs and cannot know selectors: network idle with a hard timeout, then grab whatever rendered.
  • You just clicked, scrolled, or typed and need the UI to settle: short fixed delay or a wait for the resulting element, whichever is easier.
  • Belt and suspenders for critical pages: combine them. Wait for network idle first, then confirm the selector exists before extracting.

Whatever you choose, log timeouts separately from successes. A wait that times out but still returns HTML is your early-warning system that a site changed underneath you.

Or Skip the Wait Tuning Entirely

Tuning waits per site, and re-tuning them every time a site redeploys, is the ongoing tax of running your own browsers. It sits on top of the other costs covered in curl vs headless vs stealth browsers: memory-hungry browser fleets, fingerprint patching, and proxy management.

link.sc handles the wait logic for you. It is one API to fetch and search anything online, built for LLMs and agents. Send a URL, and it escalates cheapest-first: plain HTTP with real browser TLS fingerprints, then a stealth browser with smart waits when the page needs JavaScript. It learns which method each domain requires, so repeat fetches skip straight to what works, and you get back clean markdown, JSON, raw HTML, or a screenshot of the fully rendered page.

curl -X POST https://api.link.sc/v1/fetch \
  -H "x-api-key: lsc_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/products", "format": "markdown"}'

No selectors to maintain, no idle windows to tune, no sleeps to guess. If a page renders client-side, you still get the content that a correctly-waited browser would see, without operating the browser.

Tired of guessing how long to wait? Get a free link.sc API key with 500 free credits a month and fetch fully rendered pages with one call.