How to Scrape Data Tables, Including JavaScript-Rendered Ones

Quick answer: For a static HTML table, pandas.read_html(url) reads it into a DataFrame in one line. For a JavaScript-rendered table, load the page in a real browser, wait for the rows to appear, then extract. Best of all, when a table is populated by an XHR call, skip the DOM entirely and request that API for clean JSON. Once you have the data, export it with df.to_csv or work with it directly as a DataFrame.

Tables are where the good data lives: prices, standings, financials, inventory. They are also where scrapers quietly break, because a table that looks static in your browser is often filled in by JavaScript after load. This guide covers the three cases in order of increasing effort, and the golden rule is to try the easy one first.

Case 1: Static HTML Tables (One Line)

If the table markup is present in the raw HTML, pandas reads it directly. No parser, no loop.

import pandas as pd

tables = pd.read_html("https://en.wikipedia.org/wiki/List_of_largest_companies")
df = tables[0]   # read_html returns a list of every table on the page
print(df.head())

read_html returns a list because a page can have many tables. Print len(tables) and index the one you want. This works whenever the <table>, <tr>, and <td> tags exist in the downloaded HTML.

To check before you write code: view the page source (not the rendered DOM) and search for <table. If the rows are there, read_html will find them.

If pandas struggles with the raw request (some sites block the default agent), fetch the HTML yourself and pass the string:

import requests, pandas as pd

html = requests.get(url, headers={
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
}).text
df = pd.read_html(html)[0]

Case 2: JavaScript-Rendered Tables (Wait, Then Extract)

If the source HTML has an empty <table> or no table at all, the rows are rendered client-side. read_html on the raw HTML returns nothing useful. You need to render the page first, wait for the rows, then read the DOM.

from playwright.sync_api import sync_playwright
import pandas as pd

def scrape_js_table(url):
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto(url, wait_until="domcontentloaded")

        # Wait for the rows to actually exist, do not guess with sleep
        page.wait_for_selector("table tbody tr")

        # Hand the rendered HTML to pandas
        html = page.content()
        browser.close()

    return pd.read_html(html)[0]

df = scrape_js_table("https://example.com/live-standings")
print(df.head())

The important line is wait_for_selector("table tbody tr"). It blocks until at least one data row exists, which is far more reliable than a fixed sleep. If you want the finer points of waiting, the Playwright web scraping guide covers every wait strategy.

If the columns are messy, extract cell by cell instead of trusting read_html:

rows = page.evaluate("""
    () => Array.from(document.querySelectorAll('table tbody tr')).map(tr =>
        Array.from(tr.querySelectorAll('td')).map(td => td.innerText.trim())
    )
""")
df = pd.DataFrame(rows, columns=["rank", "team", "points"])

Case 3: Tables Loaded via XHR (Hit the API)

This is the best case, and people skip it because the browser approach feels obvious. Most dynamic tables are filled by a background request that returns JSON. If you find it, you get clean, typed data with no rendering and no parsing.

Open DevTools, go to the Network tab, filter to Fetch/XHR, and reload or sort the table. Look for a request whose response is the table data as JSON. Then call it directly.

import requests, pandas as pd

resp = requests.get(
    "https://example.com/api/standings",
    params={"season": 2026, "limit": 100},
    headers={
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
        "Accept": "application/json",
        "Referer": "https://example.com/standings",  # some APIs require this
    },
    timeout=20,
)
df = pd.DataFrame(resp.json()["rows"])
print(df.head())

This is faster, more stable, and less likely to break when the site changes its CSS. When there is a usable API, always prefer it. The same network-first instinct powers extracting structured JSON from any webpage.

Comparing the Three Cases

Table type	Tell	Tool	Effort
Static HTML	<table> rows in page source	pandas.read_html	Lowest
JS-rendered	Empty table in source, rows in browser	Playwright then read_html	Medium
XHR-loaded	Rows arrive in a Fetch/XHR call	requests on the API	Low, once found

Work top to bottom. Try read_html first, check the Network tab for an API second, and only reach for a full browser when neither shortcut exists.

Handle Pagination and Sorting State

Big tables page and sort, and both change the request. Sorting usually adds a query param (?sort=points&dir=desc) to the same API. Pagination adds page or offset. Reproduce whatever you see in the Network tab, and loop with a cap.

frames = []
for page in range(1, 21):  # hard cap
    data = requests.get(api, params={"page": page, "sort": "points"}).json()
    rows = data.get("rows", [])
    if not rows:
        break
    frames.append(pd.DataFrame(rows))

df = pd.concat(frames, ignore_index=True).drop_duplicates(subset="id")

The drop_duplicates matters because rows can shift between pages while you crawl. For the full set of pagination patterns, see how to scrape paginated listings.

Export to CSV or Keep as a DataFrame

Once the data is in a DataFrame, exporting is trivial.

df.to_csv("table.csv", index=False)          # CSV
df.to_json("table.json", orient="records")   # JSON records
df.to_excel("table.xlsx", index=False)       # needs openpyxl

# Or clean it up before saving
df["points"] = pd.to_numeric(df["points"], errors="coerce")
df = df.dropna(subset=["team"])

Coercing numeric columns is worth doing early, because scraped cells arrive as strings and quiet type bugs are the most common reason a later calculation is wrong.

Let a Fetch API Render the Page

When you would rather not run a browser just to get a rendered table, a fetch API renders the page server-side and returns clean content you can parse or read as markdown:

curl https://link.sc/v1/fetch \
  -H "Authorization: Bearer lsc_..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/live-standings", "format": "markdown"}'

Markdown tables come back as clean pipe-delimited rows you can feed straight into pandas or an LLM. Options are in the link.sc docs.

A Note on Ethics

Scrape public tables only, check robots.txt, respect documented rate limits, and add a delay between requests. When a site publishes the same data through an official API or a CSV download, use it: it is more stable and it is the access the operator intended.

Wrapping Up

Tables come in three flavors, and the right tool depends on where the rows live. Try pandas.read_html first, check the Network tab for a JSON API second, and render with a browser only when nothing else works. Coerce your types, dedupe across pages, and export to whatever the next step needs.

---

Wrangling tables out of JavaScript-heavy pages? link.sc renders any URL and returns clean markdown or JSON tables in one call. Get started free with 500 credits a month.