How to Scrape Infinite Scroll and Lazy-Loaded Pages

Quick answer: The best way to scrape infinite scroll is to skip the scrolling entirely: open your browser's network tab, find the XHR request that loads each batch of items, and call that API directly with incrementing page or cursor parameters. If there's no usable API, automate scrolling in a headless browser and wait for new content to appear before extracting. Rendering APIs can also handle the scroll-and-wait for you.

Infinite scroll exists for humans on phones, and it's actively hostile to scrapers: the initial HTML contains twenty items, and the other thousand only exist after JavaScript fetches them in batches as you scroll.

The good news is that the mechanism that makes it annoying also makes it beatable, because those batches have to come from somewhere.

Technique 1: Find the Underlying API (Do This First)

Every infinite scroll is powered by a background request. The page JavaScript calls some endpoint, gets JSON, and renders it. Your job is to find that endpoint and become its client.

The recipe:

1. Open DevTools, Network tab, filter to Fetch/XHR.
2. Clear the log, then scroll the page until new items load.
3. Look at the request that just fired. Its response is your data, already structured, no HTML parsing required.
4. Study how the next batch is requested: usually page=2, offset=40, or cursor=abc123 in the URL or POST body.

Then replicate it:

import requests, time

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
    "Accept": "application/json",
    # Some APIs check these; copy from the real request in DevTools
    "Referer": "https://example.com/products",
})

items, cursor = [], None
while True:
    params = {"limit": 40}
    if cursor:
        params["cursor"] = cursor
    data = session.get("https://example.com/api/products",
                       params=params, timeout=15).json()
    items.extend(data["items"])
    cursor = data.get("nextCursor")
    if not cursor:
        break
    time.sleep(1)

print(f"Collected {len(items)} items")

This is faster than any browser approach by an order of magnitude, returns clean JSON, and doesn't break when the CSS changes. When people ask me how to scrape a lazy-loading site, "did you check the network tab?" solves it more than half the time.

Know the pagination patterns you'll meet:

Pattern	Looks like	How to iterate
Page number	?page=3	Increment until an empty response
Offset/limit	?offset=80&limit=40	Add limit to offset each call
Cursor	?cursor=eyJpZCI6...	Use the cursor from each response

Cursor pagination is the most common on modern sites and the least guessable: you must take each cursor from the previous response, you can't fabricate it.

The catch: some APIs require auth tokens, signed headers, or session cookies that are painful to replicate outside the browser. If an hour of header-copying hasn't produced a working request, stop and move to technique 2.

Technique 2: Automate the Scroll in a Headless Browser

When the API is locked down, do what a user does, just programmatically. Playwright is my pick for this:

from playwright.sync_api import sync_playwright

def scrape_infinite_scroll(url, max_rounds=30):
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page()
        page.goto(url, wait_until="domcontentloaded")
        page.wait_for_selector(".product-card")

        prev_count = 0
        for _ in range(max_rounds):
            page.mouse.wheel(0, 12000)
            try:
                # Wait until the item count actually grows
                page.wait_for_function(
                    f"document.querySelectorAll('.product-card').length > {prev_count}",
                    timeout=8000,
                )
            except Exception:
                break  # no new items: we've hit the bottom
            prev_count = page.locator(".product-card").count()

        cards = page.locator(".product-card")
        items = [{
            "title": cards.nth(i).locator("h3").inner_text(),
            "price": cards.nth(i).locator(".price").inner_text(),
        } for i in range(cards.count())]
        browser.close()
        return items

The details that separate this from the naive version you'll find on Stack Overflow:

• A max rounds cap. Truly infinite feeds will scroll forever; decide your stopping point up front.
• Growth-based exit. Stop when scrolling stops producing new items, not after a fixed number of scrolls.
• Watch for virtualization. Some lists (React virtualized lists especially) remove items that scroll out of view, so the DOM never holds more than 50 at once. If your count plateaus while the scrollbar keeps shrinking, extract items incrementally during the loop instead of once at the end.
• "Load more" buttons. Some lazy loaders want a click, not a scroll. Same loop, but page.click("button:has-text('Load more')") instead of mouse.wheel.

Waiting Correctly: The Difference Between Flaky and Solid

Most lazy-load scrapers fail not at scrolling but at waiting. The page said it loaded, your selector came back empty, and the data arrived 800ms after you gave up.

Rules that will save you hours:

• Never use fixed sleeps as your primary wait. time.sleep(3) is both too slow on fast pages and too fast on slow ones.
• Wait for the thing you want. wait_for_selector(".product-card") beats waiting for networkidle, which modern sites with analytics beacons may never reach.
• Wait for conditions, not moments. wait_for_function on an item count, as above, expresses "wait until more content exists," which is what you actually mean.
• Set timeouts you can act on. A timeout means the page changed or blocked you; log it loudly instead of silently returning partial data.

This is the same discipline that applies to all client-rendered sites, and the broader toolkit is in scraping JavaScript-rendered websites. And however you fetch, batch endpoints are easy to hammer, so keep delays between rounds; the etiquette in ethical web scraping best practices applies double here.

Technique 3: Let a Rendering API Do the Scroll-and-Wait

Running headless browsers in production is its own job: they're memory-hungry, they crash, they get fingerprinted, and every site needs its own wait logic tuned.

A rendering API moves that whole problem server-side. With link.sc, a fetch renders the page in a real browser environment and returns the settled content as clean markdown:

curl https://link.sc/v1/fetch \
  -H "Authorization: Bearer lsc_your_key" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/products", "format": "markdown"}'

Lazy-loaded content that appears on render is included; you get what a patient user with a real browser would see, without operating the browser fleet yourself. The quickstart has the details, and the free tier is enough to test whether it captures the content you need before you commit to anything.

My Decision Order

1. Network tab first, always. Ten minutes of inspection can replace a hundred lines of browser automation.
2. Rendering API second if the API is locked down and you'd rather not run browsers.
3. Your own Playwright loop third, when you need fine-grained control over exactly how far to scroll and what to extract mid-flight.

The mistake is starting at step 3 because it feels like "real" scraping. The data was sitting in a JSON endpoint the whole time.

---

Tired of tuning scroll loops? Sign up for link.sc and fetch dynamic pages as clean markdown with one API call.