Quick answer: You can scrape data behind a login only when you are authorized to access that data, for example your own account or data you have explicit permission to collect. In those cases you authenticate the way a browser does (log in, capture the session cookie, and send it with your requests). You should not bypass authentication, share or reuse someone else's credentials, or scrape logged-in data in violation of a site's terms of service. Public data is almost always the safer path.
This is the question where the honest answer matters more than the code. "How do I scrape a page behind a login" has a technical answer and a legal one, and the legal one comes first, because getting it wrong can mean a banned account at best and a lawsuit at worst.
Let me walk through the boundaries before the mechanics.
Start with the legal picture
Two things govern whether logged-in scraping is acceptable: the site's terms of service, and the law where you and the site operate.
Most sites' terms of service explicitly prohibit automated access, scraping, or "accessing the service by any means other than the provided interface." When you created your account, you agreed to those terms. Violating them is a breach of contract, and it is grounds for the site to terminate your account and, in some cases, pursue you.
In the United States, the Computer Fraud and Abuse Act (CFAA) is the statute that comes up most. The important distinction courts have drawn is between data that is publicly available and data that sits behind an authentication barrier. Accessing a system in a way that exceeds the authorization you were given is where CFAA risk lives. Scraping public pages is treated very differently from circumventing a login to reach data you were never granted.
This post is not legal advice, and the details vary by jurisdiction and by case. For a fuller treatment of the public-versus-private line and the major cases, read is web scraping legal. If real money or real risk is on the line, talk to a lawyer.
When logged-in scraping is legitimate
There are genuinely fine reasons to automate access to data behind a login. The common thread is authorization.
| Scenario | Generally OK? | Why |
|---|---|---|
| Exporting your own data from a service | Yes | It is your data and your account |
| Automating a workflow in an account you own | Usually | You are authorized; check ToS for automation limits |
| A client asks you to collect from their own accounts | Yes, with a written scope | They authorize access to their data |
| A site offers an official API for the data | Use the API | It is the sanctioned path |
| Reusing someone else's login to reach their data | No | Not your authorization to give |
| Bypassing a paywall or auth check | No | Circumventing the barrier itself |
| Credential stuffing or guessing logins | No | Unauthorized access, likely illegal |
If your use case is in the top half of that table, the mechanics below are for you. If it is in the bottom half, stop. No technique in this article will make it acceptable, and I am not going to provide one.
The mechanics for authorized cases
When you log into a site, the server sets a session cookie (or returns a token) that identifies you on subsequent requests. Scraping your own logged-in data means holding onto that session and sending it with each request, exactly as your browser does.
Option 1: Reuse a session with requests.Session
For sites with a simple form login and no heavy JavaScript, a session object handles the cookie jar for you.
import requests
session = requests.Session()
# Log in with credentials you own and are authorized to use.
session.post(
"https://example.com/login",
data={"username": "[email protected]", "password": "your-own-password"},
timeout=20,
)
# The session now carries the auth cookie automatically.
resp = session.get("https://example.com/account/orders", timeout=20)
print(resp.text)
Keep credentials in environment variables or a secrets manager, never hardcoded and never committed to a repo.
Option 2: Log in with a headless browser
Sites that render the dashboard client-side, or that use multi-step or single-sign-on flows, are easier to drive with a real browser. Playwright can log in, then reuse the authenticated state.
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
page.goto("https://example.com/login")
page.fill("#email", "[email protected]")
page.fill("#password", "your-own-password")
page.click("button[type=submit]")
page.wait_for_url("https://example.com/dashboard")
# Save the authenticated session for reuse across runs.
page.context.storage_state(path="auth.json")
print(page.content())
browser.close()
Saving storage_state lets later runs skip the login step, which means fewer authentication requests and less load on the site. If the site sends the dashboard data as JSON in the background, the same session cookie usually works against that endpoint too. The technique for spotting it is in our guide to finding hidden API endpoints, applied only to your own authorized data.
Why public alternatives are usually better
Even when logged-in scraping is permitted, it is fragile and higher-risk. Sessions expire, sites add multi-factor prompts, and an account flagged for unusual activity can be locked. Whenever the data you need also exists in public form, prefer that path:
- Official APIs and exports. Many services offer a data export or a developer API. Slower to set up, but sanctioned and stable.
- Public pages. If the information appears on a page any anonymous visitor can load, scrape that page instead, politely.
- Licensed data providers. For firmographic or market data, a licensed feed is often cheaper than maintaining brittle logged-in scrapers.
For public content, link.sc fetches any URL to clean markdown or JSON without you managing browsers or proxies:
curl https://link.sc/v1/fetch \
-H "Authorization: Bearer lsc_..." \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/public-listing", "format": "markdown"}'
link.sc is built for public web data. It is not a tool for bypassing authentication, and you should not point it at pages that require a login you are not entitled to use.
An ethics note
The rule I hold to is simple: automation should not let you reach data a person with your same authorization could not reach by hand. If you can log into an account you own and see the data, automating your own access is usually fine. If reaching the data requires borrowing a credential, defeating an auth check, or exceeding what you were granted, the technique is not the problem: the access is.
Respect terms of service, respect robots.txt, keep request rates human, and when in doubt, ask the site for permission or use their official API. The compliant path is slower, and it is the one that does not end with a legal letter.
Need clean, structured data from public web pages without the compliance headache? Try link.sc free.