Playwright Web Scraping: A Complete Guide With Runnable Code

Quick answer: Playwright is a browser automation library that drives a real Chromium, Firefox, or WebKit engine, which makes it the right tool for scraping JavaScript-heavy pages that a plain HTTP request returns empty. Install it with pip install playwright then playwright install, launch a browser, navigate to the URL, wait for the content you care about with page.wait_for_selector, and read the data off the rendered DOM. For pages that render server-side or when you just want clean markdown back, a fetch API is faster and cheaper than running a browser.

I reach for Playwright when a page only makes sense after its JavaScript runs. If you curl a modern React or Vue app you often get a near-empty shell with a <div id="root"> and nothing inside it. Playwright loads that page in a real engine, runs the scripts, and hands you the DOM a human would see.

This guide is the practical version: install it, drive it, wait correctly, extract cleanly, and know when to stop using it.

Install Playwright

Playwright ships browser binaries separately from the package, so installation is two steps.

pip install playwright
playwright install chromium

The second command downloads the browser engine. You can install all three engines with a bare playwright install, but Chromium alone covers most scraping.

Node users get the same tool:

npm install playwright
npx playwright install chromium

I will use Python for the examples. The API is nearly identical across languages.

Launch a Browser and Navigate

Here is the smallest useful script. It launches Chromium, opens a page, and prints the title.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com", wait_until="domcontentloaded")
    print(page.title())
    browser.close()

A few things worth knowing right away:

• headless=True runs with no visible window. Set it to False while developing so you can watch what the browser actually does. That single change solves half of all scraping confusion.
• wait_until controls when goto returns. "domcontentloaded" fires early, "load" waits for images and stylesheets, and "networkidle" waits until the network goes quiet. For dynamic pages I usually pair a fast wait_until with an explicit wait for a specific element (next section).

Wait Strategies (The Part Everyone Gets Wrong)

The number one Playwright mistake is extracting data before the page has rendered it. The fix is never time.sleep. Wait for a condition instead.

# Wait for a specific element to exist in the DOM
page.wait_for_selector("div.product-card")

# Wait for the network to settle (good after triggering a fetch)
page.wait_for_load_state("networkidle")

# Wait for a custom condition
page.wait_for_function("document.querySelectorAll('.product-card').length > 10")

Playwright also has auto-waiting built into its actions. When you call page.click("button.load-more"), it waits for that button to be visible and actionable first. You get a lot of reliability for free, which is why I prefer it over older tools.

Here is the difference in practice:

Approach	Reliability	Speed	Notes
time.sleep(5)	Poor	Slow	Guesses; too short breaks, too long wastes time
wait_for_selector	High	Fast	Returns the moment the element appears
wait_for_load_state("networkidle")	Medium	Medium	Good for XHR-driven content; can hang on sites that poll
wait_for_function	High	Fast	Best for "wait until N items loaded"

Extract Data From the Rendered Page

Once the content is there, you read it with locators. A locator is a lazy, re-queryable handle to elements.

# Single value
price = page.locator("span.price").inner_text()

# All matches
titles = page.locator("h2.title").all_inner_texts()

# An attribute
href = page.locator("a.detail-link").first.get_attribute("href")

For structured extraction across many cards, loop over the locator count:

cards = page.locator("div.product-card")
results = []
for i in range(cards.count()):
    card = cards.nth(i)
    results.append({
        "title": card.locator("h2").inner_text(),
        "price": card.locator(".price").inner_text(),
        "url": card.locator("a").get_attribute("href"),
    })

If you would rather run one big query in the browser, page.evaluate drops you into JavaScript and returns plain data:

data = page.evaluate("""
    () => Array.from(document.querySelectorAll('.product-card')).map(c => ({
        title: c.querySelector('h2')?.innerText,
        price: c.querySelector('.price')?.innerText,
    }))
""")

Handle Dynamic Content and Interactions

Real pages need clicks, scrolls, and form input before the data shows up. Playwright handles all of it, and it waits for each action to be safe.

# Click a "load more" button until it disappears
while page.locator("button.load-more").count() > 0:
    page.click("button.load-more")
    page.wait_for_load_state("networkidle")

# Fill and submit a search
page.fill("input[name='q']", "wireless headphones")
page.press("input[name='q']", "Enter")
page.wait_for_selector(".search-result")

For a deeper treatment of pre-scrape interactions, I wrote a separate walkthrough on how to click and interact before scraping.

Take Screenshots

Screenshots are useful for debugging and for capturing pages as evidence.

page.screenshot(path="page.png")            # visible viewport
page.screenshot(path="full.png", full_page=True)  # entire scrollable page
page.locator(".chart").screenshot(path="chart.png")  # one element

When a scraper "works on my machine" but returns nothing in CI, a full-page screenshot is the fastest way to see what the headless browser actually loaded (often a cookie wall or a bot check).

A Complete Runnable Example

This scrapes a paginated product listing, following the "next" link until it runs out, with a hard page cap so it can never loop forever.

from playwright.sync_api import sync_playwright
import json, time

def scrape(start_url, max_pages=10):
    results = []
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto(start_url, wait_until="domcontentloaded")

        for _ in range(max_pages):
            page.wait_for_selector("div.product-card")
            cards = page.locator("div.product-card")
            for i in range(cards.count()):
                card = cards.nth(i)
                results.append({
                    "title": card.locator("h2").inner_text(),
                    "price": card.locator(".price").inner_text(),
                })

            next_link = page.locator("a[rel='next']")
            if next_link.count() == 0:
                break
            next_link.first.click()
            page.wait_for_load_state("networkidle")
            time.sleep(1)  # be polite between pages

        browser.close()
    return results

if __name__ == "__main__":
    data = scrape("https://example.com/products")
    print(json.dumps(data, indent=2))
    print(f"scraped {len(data)} items")

Swap the selectors and the start URL for your target. The structure (wait, extract, find next, repeat, cap) is the same for most listings.

When to Use Playwright vs a Fetch API

Playwright is powerful, but a full browser per page is heavy. You pay in memory, CPU, and maintenance: every browser upgrade, every proxy rotation, every bot check becomes your problem. That cost is worth it when you genuinely need interaction or engine-level rendering.

When you just need the content of a page as clean text, a fetch API does the rendering for you and returns markdown or JSON in one HTTP call. link.sc handles the browser, proxies, and parsing server-side:

curl https://link.sc/v1/fetch \
  -H "Authorization: Bearer lsc_..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/products", "format": "markdown"}'

That is one request instead of a browser fleet. See the link.sc docs for the full parameter set.

My rule of thumb:

Situation	Use
Multi-step interaction, logins, complex clicks	Playwright
JS-rendered page, you want clean text back	Fetch API
Static HTML you can parse yourself	requests + a parser
Scale across thousands of URLs without babysitting browsers	Fetch API

For a broader comparison of the three tiers, see the wider guide to scraping JavaScript-rendered websites.

A Note on Ethics

Scrape public data, respect robots.txt, keep your request rate reasonable, and do not use automation to get past login walls you are not authorized to cross. Adding a short delay between pages, as in the example above, is both polite and less likely to get you blocked.

Wrapping Up

Playwright earns its place when a page needs a real browser: install it, launch headfully while developing, wait for conditions instead of sleeping, and extract with locators. When you only need the rendered content and not the interaction, skip the browser fleet and let a fetch API do the heavy lifting.

---

Tired of maintaining headless browsers just to read a page? link.sc renders any URL and returns clean markdown or JSON in one call. Start free with 500 credits a month.