← All posts

How to Rotate User Agents and Headers Correctly (Without Getting Flagged)

GUIDES · JUL 18, 2026 · 5 MIN READ

Quick answer: Rotating the User-Agent string alone does not work and often makes you easier to catch. Modern detection checks whether your whole request is internally consistent: the UA, the Accept, Accept-Language, and Sec-CH-UA headers, their order, and the underlying TLS fingerprint all have to match the browser you claim to be. Rotate coherent header sets that correspond to real browser versions, keep them consistent within a session, and make sure your TLS layer agrees with the story your headers tell.

The single most common scraping mistake I see is a script that picks a random User-Agent from a list of fifty strings on every request and calls it "stealth." It is not stealth. It is a signal. Let me explain why, and what to do instead.

Why a rotating User-Agent alone fails

A User-Agent string is a claim: "I am Chrome 126 on Windows." Detection systems check whether the rest of the request backs up that claim. Three ways a naive rotator gives itself away:

Header inconsistency. Real Chrome sends a specific Accept value, an Accept-Language, an Accept-Encoding that includes br, and a set of Sec-CH-UA client-hint headers whose contents match the version in the UA. If your UA says Chrome 126 but you send no client hints and a Python-default Accept: */*, the claim and the evidence disagree.

Header order. Browsers send headers in a stable, characteristic order. HTTP libraries send them in a different order, often alphabetical or insertion order. A server that inspects the raw order sees a fingerprint no browser produces.

TLS fingerprint mismatch. This is the big one. Before a single header is sent, your TLS client sends a ClientHello with a specific cipher list, extension order, and settings. That produces a JA3/JA4 fingerprint. Python's requests, Go's default client, and Node all have fingerprints that no browser has. So you can send a flawless Chrome header set and still get flagged, because the handshake underneath announced "I am a Python script" before Chrome was ever mentioned. I go deep on this in TLS fingerprinting and bot detection.

The lesson: a UA is one line in a story, and the story has to be consistent from the TLS handshake up.

What a coherent header set looks like

Instead of one random string, rotate whole sets where every field matches a real browser build. Here is a Chrome-on-Windows set and a Safari-on-macOS set, kept internally consistent.

CHROME_WIN = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept": ("text/html,application/xhtml+xml,application/xml;q=0.9,"
               "image/avif,image/webp,image/apng,*/*;q=0.8"),
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br, zstd",
    "Sec-CH-UA": '"Chromium";v="126", "Google Chrome";v="126", "Not.A/Brand";v="24"',
    "Sec-CH-UA-Mobile": "?0",
    "Sec-CH-UA-Platform": '"Windows"',
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Upgrade-Insecure-Requests": "1",
}

SAFARI_MAC = {
    "User-Agent": (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 "
        "(KHTML, like Gecko) Version/17.4 Safari/605.1.15"
    ),
    "Accept": ("text/html,application/xhtml+xml,application/xml;q=0.9,"
               "*/*;q=0.8"),
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    # Safari does not send Sec-CH-UA client hints. Adding them would be a tell.
}

Notice what is not in the Safari set: no Sec-CH-UA headers, because Safari does not send them. Copying Chrome's client hints onto a Safari UA is exactly the kind of inconsistency detectors look for. Each set has to be true to the browser it names.

Rotate sessions, not individual requests

Here is a counterintuitive rule: within a single browsing session, a real user does not change browsers. So flipping your header set on every request is itself unnatural. A human loads a page, then loads its images, CSS, and API calls all as the same browser.

Rotate at the session boundary, not the request boundary. Pick one coherent set per session, keep it for all requests in that session, and switch sets when you start a new session (new IP, new cookie jar, new logical user).

import random, requests

HEADER_SETS = [CHROME_WIN, SAFARI_MAC]  # add more real builds

def new_session():
    s = requests.Session()
    s.headers.update(random.choice(HEADER_SETS))
    return s

# One session = one consistent identity for its lifetime
session = new_session()
r1 = session.get("https://example.com/")
r2 = session.get("https://example.com/page-2")  # same identity, correct

Also set Referer sensibly. A request to a deep page with no Referer and Sec-Fetch-Site: none looks like someone typed the URL directly, which is fine occasionally but suspicious for every request in a crawl. When you follow a link, set the Referer to the page you came from.

When rotation helps, and when it hurts

Situation Rotate headers? Why
High-volume crawl across many pages Yes, per session Spreads load, mimics diverse real traffic
Logged-in session with cookies No Switching identity mid-session breaks the session and flags you
Site that fingerprints TLS Rotation alone will not help Fix the TLS layer first
Small, polite, low-volume fetch Optional One honest, consistent UA is often fine
Public API with an API key No, identify yourself Use the documented client identity

Rotation is a tool for blending into diverse legitimate traffic, not a magic cloak. If a site is blocking you on TLS fingerprint or IP reputation, no amount of header shuffling fixes it. Match the problem to the fix.

Fixing the TLS layer

To make your handshake match your headers, use a client that impersonates a browser's TLS stack. In Python, curl_cffi wraps curl-impersonate and produces a real Chrome JA3.

from curl_cffi import requests as cffi

# impersonate="chrome" makes the TLS handshake match the Chrome UA
r = cffi.get(
    "https://example.com/",
    headers=CHROME_WIN,
    impersonate="chrome126",
    timeout=30,
)
print(r.status_code)

Now the handshake and the headers tell the same story. That is the whole game: consistency, top to bottom.

Let the server side handle it

Maintaining accurate header sets is a chore because browser versions move and client hints evolve. Every time Chrome ships a major version, your "current" set is slightly stale, and stale is detectable.

With link.sc, the header sets, TLS impersonation, and rotation are maintained server side and kept current, so you never ship a Chrome 126 header set from a TLS stack that looks like Python.

curl https://link.sc/v1/fetch \
  -H "Authorization: Bearer lsc_..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/", "format": "markdown"}'

You get clean content back and never touch a Sec-CH-UA string. See the link.sc docs for the full fetch options.

A note on ethics

Coherent headers are for accessing public content the way a normal browser would, not for defeating authentication or scraping data you are not permitted to. Respect robots.txt and rate limits, identify your bot where a site asks you to, and do not impersonate a browser to get around a paywall or a login. Blending in is fine; breaking in is not.

Stop chasing browser-version header sets by hand: link.sc keeps headers and TLS fingerprints current for you. Try it free.