Quick answer: To interact with a page before scraping, drive it with a headless browser like Playwright: locate the element, call .click() or .fill(), then wait for the resulting change before reading the DOM. This handles cookie banners, "load more" buttons, and forms. The reliable pattern is always click, wait for a signal, then scrape, never click and immediately read.
Plenty of pages will not give up their content until you do something first. A cookie wall covers everything. A "load more" button hides half the list. A search form gates the results. Static fetching cannot deal with any of that. You need to automate the interaction.
Here is how to click, dismiss, and fill your way to the content you actually want.
When you need interaction at all
Before you reach for a browser, check whether you actually need one. A lot of "I have to click load more" situations are really a paginated API in disguise. Open DevTools, watch the Network tab while you click, and see if a clean JSON request fires. If it does, hit that directly and skip the browser. Our post on scraping React and SPA sites covers finding that hidden API.
If the content genuinely only appears after an interaction with no clean API behind it, automate the browser.
Dismissing cookie and consent popups
Consent modals are the most common blocker. They overlay the page and sometimes block clicks on everything underneath. The fix is to click the accept or close button, then wait for the modal to disappear.
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
page.goto("https://example.com")
# Click accept if the banner is present, ignore if it is not
accept = page.get_by_role("button", name="Accept")
if accept.is_visible():
accept.click()
page.wait_for_selector(".cookie-banner", state="hidden")
content = page.content()
browser.close()
Two things make this robust. First, checking is_visible() so the script does not crash on pages without a banner. Second, get_by_role("button", name="Accept"), which finds the button by its accessible role and text rather than a brittle CSS class. Text-based locators survive redesigns better than div.cookie-modal__btn--primary.
Clicking "load more" and infinite scroll
Lists that grow on click or scroll need a loop: act, wait for new items, repeat until nothing new appears.
prev_count = 0
while True:
cards = page.locator(".product-card")
count = cards.count()
if count == prev_count:
break # no new items loaded, we are done
prev_count = count
load_more = page.get_by_role("button", name="Load more")
if not load_more.is_visible():
break
load_more.click()
page.wait_for_function(
"(n) => document.querySelectorAll('.product-card').length > n",
arg=count,
)
For infinite scroll, replace the click with a scroll and the same wait-for-more logic.
page.mouse.wheel(0, 20000)
page.wait_for_timeout(500) # brief settle, then re-check the count
Always add a stop condition (count stops growing, or a max iteration cap). An unbounded loop on infinite scroll will run forever.
Filling and submitting forms
Search boxes and filters are just form fills followed by a submit and a wait for results.
page.fill("input[name='q']", "wireless headphones")
page.get_by_role("button", name="Search").click()
page.wait_for_selector(".search-results .result")
results = page.locator(".search-results .result h3").all_inner_texts()
The pattern is the same every time: fill, submit, wait for the result container, then read.
The rule that matters: wait after every action
The single most common bug in interaction scraping is reading the DOM before the action's effect has landed. A click kicks off an async request; the content is not there yet when the next line runs.
Never do this:
load_more.click()
cards = page.locator(".product-card").all_inner_texts() # too early, misses new items
Do this instead:
| After you | Wait for |
|---|---|
| Click "accept" on a modal | The modal to be hidden |
| Click "load more" | The item count to increase |
| Submit a search | The results container wait_for_selector |
| Scroll for more | A short settle, then re-check count |
| Navigate to a new page | wait_until="networkidle" or a selector |
Waiting on a state change (a selector appearing, a count growing) beats wait_for_timeout() every time, because it finishes as soon as the change happens and does not silently pass when the change never comes.
Selector stability: make your clicks last
Interaction scripts break when selectors change. Some locators are far more durable than others.
| Locator type | Example | Durability |
|---|---|---|
| Accessible role + name | get_by_role("button", name="Accept") |
High |
| Visible text | get_by_text("Load more") |
High |
| Test id | get_by_test_id("submit") |
High if the site uses them |
| Stable id | #search-submit |
Medium |
| Generated CSS class | .btn-xh39fk |
Low, breaks on rebuild |
| Deep XPath | /html/body/div[3]/div[2]/button |
Very low |
Prefer role and text locators. They mirror how a human finds the button, and humans still find the accept button after a redesign that changes every class name.
An important ethics note
This is where you have to be careful. Automating clicks and forms is fine for public content: dismissing a cookie banner, loading more public listings, running a public search. It is not fine for getting past authentication.
Do not automate logins to scrape private data, do not fill credentials to bypass a paywall, and do not click through a consent flow just to defeat access controls the site put there on purpose. If content sits behind a login for a reason, respect that. Stay on the public side of the line, honor robots.txt and the site's terms, and rate-limit your interactions so you are not hammering the server. Interaction automation is powerful, which is exactly why it is easy to misuse.
When to skip the browser
Driving a real browser is heavy, and interaction logic is fragile by nature. If you want to decide when a browser is worth it at all, our breakdown of curl vs headless vs stealth browser covers the tradeoffs. If the content is reachable through the JSON API behind those buttons, use it. And if you just need the rendered content of a page (after the obvious modals) without writing and maintaining click scripts, a fetch API that renders pages handles the common cases for you and returns clean content:
curl https://link.sc/v1/fetch \
-H "Authorization: Bearer lsc_..." \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "format": "markdown"}'
link.sc renders the page and returns clean markdown, so you only script interactions for the genuinely complex flows. The fetch docs cover the options.
Bottom line
Interacting before scraping is a small, repeatable pattern: locate with a durable selector, act, wait for the effect, then read. Playwright makes each step one line. Keep your selectors role and text based so they survive redesigns, add stop conditions to your loops, and stay firmly on the public side of any auth wall.
Skip the browser scripting for the common cases: fetch rendered, clean content from any public page in one call. Try link.sc free.