← All posts

How to Scrape Paginated Listings and Search Results

GUIDES · JUL 18, 2026 · 6 MIN READ

Quick answer: To scrape a paginated listing, first identify which pagination pattern it uses: page numbers (?page=2), offset and limit (?offset=40&limit=20), a cursor (?cursor=abc123), infinite scroll, or a "load more" button. Then loop through pages, always with a hard page cap and a delay, collecting and deduplicating results as you go, and stop when a page returns no new items. The cleanest approach is usually to call the underlying API the page uses rather than parsing rendered HTML.

Almost every listing worth scraping (products, jobs, search results, articles) is spread across pages. The mechanics are simple once you know which of the five patterns you are looking at. This post shows how to spot each one and gives you a safe loop for each. The theme throughout is the same: find the pattern, cap the loop, and dedupe.

First, Detect the Pattern

Before writing any loop, open your browser DevTools, go to the Network tab, filter to Fetch/XHR, and click to page 2. What you see tells you everything.

Pattern What the request looks like How you advance
Page number ?page=1, ?page=2 Increment the number
Offset / limit ?offset=0&limit=20 Add limit to offset
Cursor ?cursor=eyJpZCI6MTAwfQ Read the next cursor from the response
Infinite scroll XHR fires on scroll Same as offset or cursor, triggered by scroll
Load more XHR fires on button click Same, triggered by a click

If the data comes back as JSON in that Network tab, call that endpoint directly and skip HTML parsing entirely. That is faster, more stable, and gives you structured data for free. Only fall back to parsing rendered HTML when there is no usable API.

Pattern 1: Page Numbers

The most common pattern. The page number lives in the URL, and you increment it until a page comes back empty.

import requests, time

def scrape_pages(base_url, max_pages=50):
    session = requests.Session()
    session.headers["User-Agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
    all_items = []

    for page in range(1, max_pages + 1):
        resp = session.get(base_url, params={"page": page}, timeout=20)
        resp.raise_for_status()
        items = resp.json().get("results", [])
        if not items:
            break  # no more pages
        all_items.extend(items)
        time.sleep(1)  # be polite

    return all_items

The empty-page check is what stops you cleanly. The max_pages cap is what stops you when the site keeps returning the last page forever (some do).

Pattern 2: Offset and Limit

Here you track a running offset and add the page size each round.

def scrape_offset(base_url, limit=20, max_items=1000):
    session = requests.Session()
    all_items, offset = [], 0

    while offset < max_items:
        resp = session.get(base_url, params={"offset": offset, "limit": limit}, timeout=20)
        items = resp.json().get("results", [])
        if not items:
            break
        all_items.extend(items)
        offset += limit
        time.sleep(1)

    return all_items

Watch for one trap: if the site ignores your offset and keeps returning page one, you will collect duplicates forever. The dedupe step below and the max_items cap both protect you.

Pattern 3: Cursor

Cursor pagination does not use numbers you control. The response hands you the token for the next page, and you stop when there is no next token.

def scrape_cursor(base_url, max_pages=100):
    session = requests.Session()
    all_items, cursor, pages = [], None, 0

    while pages < max_pages:
        params = {"cursor": cursor} if cursor else {}
        data = session.get(base_url, params=params, timeout=20).json()
        all_items.extend(data.get("results", []))

        cursor = data.get("next_cursor")
        pages += 1
        if not cursor:
            break  # last page
        time.sleep(1)

    return all_items

Cursor pagination is the friendliest to scrape correctly because the server tells you exactly when to stop.

Pattern 4: Infinite Scroll

Infinite scroll is usually offset or cursor pagination triggered by scrolling. The best move is to find that XHR and treat it like Pattern 2 or 3 above, no browser required. When there is genuinely no reachable API, you automate scrolling in a real browser and wait for new items before reading them.

I wrote a dedicated deep dive on this: see scrape infinite scroll and lazy-loaded pages for the network-first approach and the scroll-automation fallback.

Pattern 5: Load More Button

Same underlying request as infinite scroll, but triggered by a click. If the button just calls an API with an incrementing offset, call that API directly. If you must drive the page, click the button in a loop until it disappears.

from playwright.sync_api import sync_playwright

def scrape_load_more(url, max_clicks=30):
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto(url, wait_until="domcontentloaded")

        for _ in range(max_clicks):
            btn = page.locator("button.load-more")
            if btn.count() == 0 or not btn.first.is_visible():
                break
            btn.first.click()
            page.wait_for_load_state("networkidle")

        items = page.evaluate("""
            () => Array.from(document.querySelectorAll('.card')).map(c => ({
                title: c.querySelector('h2')?.innerText,
                url: c.querySelector('a')?.href,
            }))
        """)
        browser.close()
    return items

For more on driving clicks safely, see click and interact before scraping.

Loop Safely: Caps, Delays, and Stop Conditions

Every loop above shares three safeguards, and they are not optional:

  1. A hard cap (max_pages or max_items) so a misbehaving site can never spin you forever.
  2. A stop condition (empty page, or no next cursor) so you quit as soon as the data ends.
  3. A delay (time.sleep(1)) so you are not hammering the server. Slower is politer and less likely to get blocked.

Always Dedupe

Pagination produces duplicates more often than you would think: items shift between pages while you crawl, offsets get ignored, or a "next" link points back to itself. Dedupe on a stable key.

def dedupe(items, key="id"):
    seen, out = set(), []
    for item in items:
        k = item.get(key) or item.get("url")
        if k in seen:
            continue
        seen.add(k)
        out.append(item)
    return out

Prefer a real ID. Fall back to the item URL. Deduping on the full record is fragile because a changed price or timestamp makes two copies of the same item look distinct.

Let a Fetch API Handle Rendering

When the listing is JavaScript-rendered and you still want per-page content parsed for you, a fetch API removes the browser work. You loop the page URLs and let the service render each one:

curl https://link.sc/v1/fetch \
  -H "Authorization: Bearer lsc_..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/products?page=2", "format": "markdown"}'

You still own the loop, the cap, and the dedupe. The rendering, proxies, and parsing move server-side. See the link.sc docs for options, and if your goal is the whole site rather than one listing, see how to crawl an entire website.

A Note on Ethics

Only crawl public listings, check robots.txt, respect any documented rate limits, and keep a delay between pages. If the site offers an official API with pagination, use it: it is more stable and it is the access path the operator intended.

Wrapping Up

Pagination looks like five different problems but it is really one: figure out how the site asks for the next batch, then reproduce that request in a capped, deduped loop with a delay. Check the Network tab first, because a JSON endpoint turns most of this into a clean API client instead of a fragile HTML scraper.

Scraping page after page and tired of maintaining renderers? link.sc fetches and renders any URL to clean markdown or JSON so your loop stays simple. Start free with 500 credits a month.