Quick answer: A user agent is a string of text your browser or HTTP client sends with every request, in the User-Agent header, that identifies the software making the request: the browser, its version, the rendering engine, and the operating system. Servers read it to decide what to send back. You can set it to anything you want, but a user agent by itself does not make an automated client look like a real browser.
I spend a lot of my time debugging why a request gets a 200 OK in the browser and a 403 Forbidden from a script. Nine times out of ten the conversation starts with the user agent, so it is worth understanding exactly what this header is and, more importantly, what it is not.
What the User-Agent Header Actually Contains
The User-Agent (UA) header is a plain-text field in an HTTP request. Here is what a current Chrome on Windows sends:
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
That looks like nonsense until you break it apart:
| Token | Meaning |
|---|---|
Mozilla/5.0 |
A historical relic. Almost every browser claims to be "Mozilla" for backward compatibility. |
(Windows NT 10.0; Win64; x64) |
The platform: Windows 10/11, 64-bit. |
AppleWebKit/537.36 (KHTML, like Gecko) |
The rendering engine lineage. |
Chrome/126.0.0.0 |
The actual browser and version. |
Safari/537.36 |
Another compatibility claim, left over from WebKit's shared history. |
The format is a loose convention, not a strict standard. That is why it reads like a museum of every browser war since the 1990s. A curl request, by contrast, sends something honest and short:
User-Agent: curl/8.4.0
And a Python requests client sends python-requests/2.31.0. The difference is the whole story: one of these looks like a person, the others announce themselves as tools.
How Sites Use the User Agent
Servers read the UA for several legitimate reasons before they read it for defensive ones.
- Content negotiation. Serving a mobile layout to a phone, a desktop layout to a laptop.
- Feature detection (legacy). Older sites branch on browser version to work around bugs.
- Analytics. Logging which browsers and platforms visit, so teams know what to support.
- Bot management. Flagging or blocking clients whose UA is a known tool string like
python-requestsor an empty UA.
That last one is why scrapers care. A default python-requests/2.31.0 header is the single easiest bot signal to catch, and many sites drop it at the door. So the first instinct is to swap in a browser UA and move on. That instinct is right, and also not nearly enough.
Why Scrapers Set and Rotate the User Agent
If you send the default client UA, you are volunteering that you are a script. Setting a realistic browser UA removes the most obvious tell. Setting it correctly matters:
curl "https://example.com/data" \
-H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
In Python:
import requests
headers = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/126.0.0.0 Safari/537.36"
)
}
resp = requests.get("https://example.com/data", headers=headers)
Rotation is the next step. If a thousand requests in an hour all carry the identical UA from the identical IP, that pattern is itself a signal, even if the string looks legitimate. Rotating across a pool of real, current UA strings spreads the fingerprint out. I wrote a full guide on doing this without shooting yourself in the foot: see how to rotate user agents and headers correctly, because rotation done carelessly creates more anomalies than it hides.
Why a User Agent Alone Does Not Disguise a Bot
Here is the part people learn the hard way. Setting a Chrome UA on a Python request does not make you Chrome. It makes you a Python client claiming to be Chrome, and modern detection is very good at catching the mismatch.
Three things have to agree, and the UA is only one of them.
| Layer | What a real Chrome sends | What a naive script sends |
|---|---|---|
| User-Agent header | Chrome 126 string | Chrome 126 string (spoofed) |
| Other headers | Accept, Accept-Language, Sec-CH-UA, Sec-Fetch-* in Chrome's exact order |
Missing or in the wrong order |
| TLS fingerprint | Chrome's specific cipher suite and extension ordering | The Python/OpenSSL fingerprint |
When your UA says "Chrome 126" but your TLS handshake says "OpenSSL via Python" and your headers are missing the Sec-CH-UA client hints, that contradiction is louder than a wrong UA ever was. Detection systems compare these layers against each other. A consistent-but-honest curl fingerprint is sometimes less suspicious than an inconsistent-but-fancy spoof.
This is the domain of TLS fingerprinting, and it is worth understanding if you want to know why your "perfect" headers still get blocked. I go deep on it in TLS fingerprinting and bot detection. The short version: the UA is the header you can see, and the fingerprint is the one you cannot, and detectors weight the invisible one heavily.
Where to Find and Verify Your User Agent
You can read the exact header a client sends by hitting an echo endpoint:
curl https://httpbin.org/user-agent
In a browser console, navigator.userAgent returns the same string JavaScript sees. If you are building a scraper, always test against an echo service first so you know precisely what you are sending, headers and all, before you blame the target site.
A Note on Ethics
Setting a user agent to access public data is normal and expected: every browser does it. The line I hold is simple. Respect robots.txt, honor rate limits, do not spoof a UA to slip past authentication or a paywall, and do not impersonate a specific named crawler like Googlebot to get treatment you are not entitled to. Identifying your bot honestly (many teams use a custom UA with a contact URL) is often the more durable choice for long-running jobs.
Let link.sc Handle the Whole Fingerprint
If your goal is the data and not a research project in browser internals, this is exactly what link.sc exists for. You send a URL, and the fetch layer manages the UA, the full header set, the TLS fingerprint, and rotation as one consistent identity, then returns clean markdown or JSON:
curl https://link.sc/v1/fetch \
-H "Authorization: Bearer lsc_..." \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/data", "format": "markdown"}'
No UA string to maintain, no header-order guesswork, no mismatched handshake. You get the page as a well-behaved browser would see it. The free tier is 500 credits a month, which is plenty to test whether the fingerprint problem you have been fighting simply goes away.
The Bottom Line
A user agent is a self-reported label. It is useful, it is the first thing detection looks at, and it is trivially spoofable, which is exactly why nobody trusts it on its own anymore. Set a realistic one, rotate a pool of them, and understand that the header is one layer of a fingerprint that has to be internally consistent. Get the invisible layers wrong and the best UA in the world will not save you.
Tired of fighting headers and fingerprints? link.sc fetches any URL as a real browser would and hands you clean data. Start free.