Cookies and Sessions in Web Scraping: What They Are and How to Handle Them

Quick answer: Cookies are small pieces of state a server tells your client to store and send back on every later request, and a session is the server-side record those cookies point to. In web scraping they matter because HTTP is stateless: without carrying cookies, every request looks like a brand-new visitor, so anything that depends on continuity (a currency choice, a consent banner, a multi-step form) resets each time. You handle them with a cookie jar or a session object that persists cookies across requests, and sometimes a CSRF token grabbed from the page. The compliance line is firm: use cookies to maintain a normal public-browsing session, not to impersonate an authenticated user you are not.

Most scraping problems that look mysterious ("the site works in my browser but my scraper gets a different page") come down to state. Your browser is quietly carrying a dozen cookies; your scraper is carrying none. This explains what that state is, how to carry it correctly, and where the ethical boundary sits.

What a Cookie Is

HTTP has no memory. Each request is independent, and the server would have no idea two requests came from the same visitor, except for cookies. The mechanism is simple:

1. You request a page. The server responds with a Set-Cookie header: Set-Cookie: session_id=abc123; Path=/; HttpOnly.
2. Your client stores that.
3. On every later request to that site, your client sends Cookie: session_id=abc123.
4. The server reads it and knows "this is the same visitor as before."

That cookie value is usually a key. The actual state (who you are, what is in your cart, which A/B variant you got) lives on the server, in a record called a session. The cookie is the claim ticket; the session is the coat.

What a Session Is (Two Meanings)

The word "session" gets used two ways, and conflating them causes confusion:

Meaning	What it is	Who owns it
Server-side session	The stored state a session cookie points to	The website
Client session object	A client that persists cookies and connection settings across requests	Your scraper

The server-side session is theirs. The client session object is a tool in your HTTP library (requests.Session in Python) that automatically stores cookies from responses and replays them on the next request, so you do not have to shuttle headers by hand. When people say "use a session for scraping," they mean the second one.

Handling Cookies With a Session Object

Here is the difference in practice. Without a session, cookies are dropped between requests:

import requests

# No state carried: the second request is a stranger again
requests.get("https://example.com/set-currency?c=EUR")
r = requests.get("https://example.com/prices")   # prices come back in USD

With a session object, the cookie jar persists automatically:

import requests

with requests.Session() as s:
    s.headers.update({"User-Agent": "MyResearchBot/1.0"})
    s.get("https://example.com/set-currency?c=EUR")   # server sets a cookie
    r = s.get("https://example.com/prices")           # cookie replayed -> EUR
    print(s.cookies.get_dict())                       # inspect the jar

The Session object holds a cookie jar. Anything the server sets, it stores; on the next request to that host, it sends them back. That single object turns a pile of stateless requests into a coherent visit.

Cookie Jars and Reuse

A cookie jar is just the container of stored cookies. Two practical patterns:

• Persist a jar to disk so a long crawl can resume without redoing the steps that established state. Save s.cookies and reload it next run.
• One jar per identity. If you run parallel workers that each need their own session, give each its own jar. Sharing one jar across workers makes them look like a single confused visitor jumping between IPs, which is a bot tell (see how sites detect and block scrapers).

import requests, pickle, pathlib

def save_jar(session, path="jar.pkl"):
    pathlib.Path(path).write_bytes(pickle.dumps(session.cookies))

def load_jar(session, path="jar.pkl"):
    p = pathlib.Path(path)
    if p.exists():
        session.cookies.update(pickle.loads(p.read_bytes()))

CSRF Tokens and Form State

Some pages, particularly anything that accepts a POST, embed a CSRF (cross-site request forgery) token: a one-time value in a hidden form field or a meta tag that must be sent back with the submission. It is a security measure so a request has to come from someone who actually loaded the form. To submit a public form (a search, a store locator), you often have to read the token off the page first, then include it.

import requests
from bs4 import BeautifulSoup

with requests.Session() as s:
    page = s.get("https://example.com/search")
    soup = BeautifulSoup(page.text, "html.parser")
    token = soup.select_one('input[name="csrf_token"]')["value"]

    result = s.post(
        "https://example.com/search",
        data={"q": "wireless headphones", "csrf_token": token},
    )
    print(result.status_code)

The session carries the cookie that the token is bound to, and the token proves you loaded the form. Both are needed; a token without the matching session cookie is rejected.

When You Actually Need Cookies

Not every scrape needs any of this. Reach for sessions when:

• The site sets a consent or region cookie before showing full content.
• A choice (currency, language, units) has to persist across pages.
• You must submit a public form that uses a CSRF token.
• The site's anti-bot layer hands out a clearance cookie the first request must obtain before content loads.

For a plain public page that returns the same HTML to everyone, a single stateless request is simpler and better. Do not add session machinery you do not need.

If cookie and header juggling becomes the bulk of your code, a managed fetch layer can carry it for you. link.sc accepts cookies and custom headers per request, so you can pass an established session through without wiring up a jar and browser yourself:

import os, requests

resp = requests.post(
    "https://api.link.sc/v1/fetch",
    headers={"x-api-key": os.environ["LINKSC_KEY"]},
    json={
        "url": "https://example.com/prices",
        "format": "markdown",
        "cookies": {"currency": "EUR", "consent": "accepted"},
    },
    timeout=60,
)
print(resp.json()["content"])

The Compliance Line

This is the section to read twice, because cookies and sessions are exactly where legitimate scraping and account abuse can blur, and the difference is not subtle.

• Public browsing state is fine. Carrying a consent cookie, a currency preference, or a bot-clearance cookie to view public content is normal browser behavior. You are being a well-behaved visitor.
• Authentication is the line. Replaying a stolen or borrowed logged-in session cookie to reach a private account, paid content, or data behind a login you were not granted is not scraping public data. It is impersonating an authenticated user, and it usually violates both terms of service and the law.
• Respect the site. Cookies do not exempt you from robots.txt or rate limits. Keep requests polite per domain, the same discipline as avoiding IP bans when scraping.

A clean test: if the cookie you are carrying belongs to a session any anonymous visitor could establish by clicking through the public site, you are fine. If it only exists because someone logged in, stop.

Cookies and sessions are just how the web remembers you between requests. Carry them with a session object, grab a CSRF token when a form demands one, and keep the whole thing on the public side of the authentication line. Do that and the "works in my browser but not my scraper" mystery mostly disappears.

---

Tired of juggling cookie jars and CSRF tokens? Get a free link.sc key and pass cookies and headers straight through to a clean fetch.