Quick answer: Selenium web scraping means driving a real browser with code to load pages, wait for content, and extract data. Install selenium, start a WebDriver for Chrome or Firefox, call driver.get(url) to navigate, locate elements with find_element and By selectors, use explicit waits so you do not read the page before it renders, then pull text and attributes. Selenium shines on JavaScript-heavy pages that plain HTTP libraries cannot see, but it is slower and easier to detect than the alternatives.
Selenium started life as a browser automation tool for testing, and scrapers borrowed it because it does one thing no HTTP library can: it runs a real browser, so JavaScript executes and the page you see is the page you scrape. This guide takes you from install to a working scraper, then tells you honestly when Selenium is the wrong tool.
Install and Set Up WebDriver
You need two things: the selenium package and a browser. Recent Selenium versions ship with Selenium Manager, which downloads the matching driver automatically, so you no longer have to hunt for chromedriver yourself.
pip install selenium
Start a browser session. Running headless (no visible window) is standard for scraping on a server.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--headless=new")
options.add_argument("--window-size=1920,1080")
driver = webdriver.Chrome(options=options)
Always close the browser when you are done, or you will leak processes.
driver.quit()
Navigate to a Page
Navigation is one call. driver.get blocks until the initial page load event fires, though not until every async script finishes, which is why waits matter later.
driver.get("https://example.com")
print(driver.title)
Locate Elements
Selenium finds elements with find_element (first match) and find_elements (all matches), paired with a By strategy.
from selenium.webdriver.common.by import By
# by CSS selector (my default)
heading = driver.find_element(By.CSS_SELECTOR, "h1")
# by ID, class, tag, or XPath
logo = driver.find_element(By.ID, "logo")
items = driver.find_elements(By.CLASS_NAME, "product")
link = driver.find_element(By.XPATH, "//a[@rel='next']")
CSS selectors cover most cases and read more cleanly than XPath. Reach for XPath when you need to select by text content or walk up the tree.
Use Waits (The Part Everyone Skips)
The single most common Selenium bug is reading an element before JavaScript has rendered it. The fix is an explicit wait: tell Selenium to poll until a condition is true, up to a timeout.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 10)
element = wait.until(
EC.presence_of_element_located((By.CSS_SELECTOR, ".product"))
)
Do not use time.sleep() for this. A fixed sleep is either too short (flaky) or too long (slow). An explicit wait returns the moment the condition is met and only waits longer when it has to.
Extract Text and Attributes
Once you have an element, pull what you need.
title = element.text # visible text
href = element.get_attribute("href") # an attribute
html = element.get_attribute("outerHTML") # raw markup
Handle Dynamic Content
Pages that load more content as you scroll, or reveal data behind a "Load more" button, need interaction. Selenium can scroll and click like a user.
# scroll to the bottom to trigger lazy loading
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
# click a button, then wait for the new content
button = driver.find_element(By.CSS_SELECTOR, "button.load-more")
button.click()
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, ".product:nth-child(20)")))
For a broader look at rendering strategies, we cover the topic in scrape JavaScript-rendered websites.
A Full Runnable Example
Here is a complete scraper that loads a page, waits for content, paginates once, and collects structured records.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
def scrape(url: str) -> list[dict]:
options = Options()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)
results = []
try:
driver.get(url)
wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, ".product")))
for card in driver.find_elements(By.CSS_SELECTOR, ".product"):
name = card.find_element(By.CSS_SELECTOR, ".name").text
price = card.find_element(By.CSS_SELECTOR, ".price").text
results.append({"name": name, "price": price})
finally:
driver.quit()
return results
if __name__ == "__main__":
rows = scrape("https://example.com/products")
for row in rows:
print(row)
The try/finally guarantees the browser closes even if a selector fails. That pattern will save you from a machine full of zombie Chrome processes.
The Honest Limits
Selenium is powerful, but it is not free of cost. Here is where it struggles.
| Concern | Reality with Selenium |
|---|---|
| Speed | Slow. Booting a full browser per page is heavy; expect seconds per page, not milliseconds. |
| Resource use | High. Each browser eats real memory and CPU, which limits concurrency. |
| Detection | Automated browsers carry signals (driver flags, missing headers) that anti-bot systems look for. |
| Flakiness | Timing bugs and stale-element errors are common without careful waits. |
| Setup at scale | Running many browsers reliably needs a grid or containers. |
When any of these bite, you have better options:
- Playwright is a more modern browser-automation library with a cleaner async API, auto-waiting built in, and generally faster runs. If you are choosing between them for a new project, start with our Selenium alternatives for scraping rundown.
- A fetch API removes the browser entirely. If you just need the rendered content of a page and do not need to click through a multi-step flow, sending a URL to a hosted service is far simpler than running Chrome. Against link.sc it looks like this:
import requests
resp = requests.post(
"https://api.link.sc/v1/fetch",
headers={"x-api-key": "lsc_your_key_here"},
json={"url": "https://example.com/products", "format": "markdown"},
timeout=60,
)
resp.raise_for_status()
print(resp.json()["content"]) # rendered page as clean markdown
The service handles rendering and returns clean markdown, so there is no WebDriver, no waits, and no browser to babysit. See the link.sc docs for options like waiting on a selector or specifying a country.
Scrape Responsibly
Selenium makes you look like a real user, which is exactly why restraint matters. Only collect public data, respect robots.txt, and do not use a browser to slip past a login or paywall. Add real delays between requests, avoid running dozens of browsers against a single small site, and identify yourself where a site asks scrapers to. Automating a browser is not permission to ignore the rules a site sets. Good behavior keeps your access working and keeps you out of legal trouble.
The Bottom Line
Selenium web scraping is the right tool when you need a real browser to run JavaScript or drive a multi-step interaction, and it is well worth learning the WebDriver, By selectors, and explicit waits. But it is slow, heavy, and detectable, so for pages you only need to read once, a lighter alternative like Playwright or a fetch API will save you time and headaches. Pick the lightest tool that gets the data.
Skip the browser farm. link.sc renders and cleans any page for you in one API call. Try it free with 500 credits a month.